Currently, it is possible to retrieve 50 or so records from Koha via OAI-PMH, when using times in your 'from' and 'until' arguments. Here is an example that will return a list of records (if your OAI server syspref is turned on). http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z However, if you need to retrieve the rest of the results via a resumption token, Koha won't return your results. http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: This is because Koha joins and splits the resumption token using colons. Obviously, since the timestamps use colons, the string isn't going to be split correctly. I propose that we change the separator colons to slashes. This is the method that DSpace uses when serving records via OAI-PMH. You can see an example here: http://papyrus.bib.umontreal.ca/dspace-oai/request?verb=ListRecords&metadataPrefix=oai_dc -- Note: Koha doesn't handle times 100% correctly in master either. That's why I opened bug 10824.
Created attachment 21633 [details] [review] Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time This patch changes the value separator in OAI-PMH resumption tokens from colons to slashes, so that the token string isn't split incorrectly when a time is included. TEST PLAN: 1) Turn on the OAI-PMH server syspref in Koha 2) Send a ListRecords request using 'from' and 'until' arguments that include times (Best to use very far apart times so that you retrieve more than 50 records which will likely be the trigger for a resumptionToken). Here is an example: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z N.B. Replace KOHAINSTANCE with the URL of your Koha instance. 3) Scroll down to the bottom of the page until you find the resumptionToken. It will look similar to this: <resumptionToken cursor="50"> oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: </resumptionToken> 4) Copy that resumption token and send a request with it like so: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: 5) The page should (incorrectly) show no records. 6) APPLY PATCH 7) Repeat steps 2, 3, and 4 8) Note that the resumptionToken now uses slashes (e.g. /) instead of colons. Note also that now the second request will show records!!! N.B. This will only happen if Koha has enough records to serve to you. If your Koha has less than 50 records, try lowering the number provided in the "OAI-PMH:MaxCount" system preference.
Patch applied cleanly, go forth and signoff
Strange, the resumptionToken request works for me, without applying the patch. It also works after I apply the patch of course.
(In reply to Petter Goksøyr Åsen from comment #3) > Strange, the resumptionToken request works for me, without applying the > patch. > > It also works after I apply the patch of course. Thanks for that Petter. I should've written my test plan better. In fact, it only looks like it's working. If you insert some debugging code, you will find that the following token: oai_dc:50:2012-09-05T13:44:33Z:2013-09-05T13:44:33Z: is being broken into the following: metadataPrefix = oai_dc offset = 50 from = 2012-09-05T13 until = 44 set = 33Z and the rest of the token will be thrown away. -- If you re-run the test, take a look at the first result you receive in the ListRecords response. Change your "until" date to before the datestamp of that first result. You will find that record still appears when it shouldn't! -- Also, the resumption token at the bottom will be similar to the following: oai_dc:66:2012-09-05T13:44:33Z This looks like a resumption token with a fully formed From date. However, it's a token built up from those split fields I identified before. It thinks 44 is the until datestamp and 33Z is the set. If you try to use that resumptionToken, you will get no results. -- Please try it again in light of these notes. I'm sure that you'll find I'm correct :)
Created attachment 22239 [details] [review] Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time This patch changes the value separator in OAI-PMH resumption tokens from colons to slashes, so that the token string isn't split incorrectly when a time is included. TEST PLAN: 1) Turn on the OAI-PMH server syspref in Koha 2) Send a ListRecords request using 'from' and 'until' arguments that include times (Best to use very far apart times so that you retrieve more than 50 records which will likely be the trigger for a resumptionToken). Here is an example: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z N.B. Replace KOHAINSTANCE with the URL of your Koha instance. 3) Scroll down to the bottom of the page until you find the resumptionToken. It will look similar to this: <resumptionToken cursor="50"> oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: </resumptionToken> 4) Copy that resumption token and send a request with it like so: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: 5) The page should (incorrectly) show no records. 6) APPLY PATCH 7) Repeat steps 2, 3, and 4 8) Note that the resumptionToken now uses slashes (e.g. /) instead of colons. Note also that now the second request will show records!!! N.B. This will only happen if Koha has enough records to serve to you. If your Koha has less than 50 records, try lowering the number provided in the "OAI-PMH:MaxCount" system preference. Signed-off-by: Petter Goksoyr Asen <boutrosboutrosboutros@gmail.com> I understand; I can now confirm the behaviour described in the test plan.
Created attachment 22450 [details] [review] [PASSED QA] Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time This patch changes the value separator in OAI-PMH resumption tokens from colons to slashes, so that the token string isn't split incorrectly when a time is included. TEST PLAN: 1) Turn on the OAI-PMH server syspref in Koha 2) Send a ListRecords request using 'from' and 'until' arguments that include times (Best to use very far apart times so that you retrieve more than 50 records which will likely be the trigger for a resumptionToken). Here is an example: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z N.B. Replace KOHAINSTANCE with the URL of your Koha instance. 3) Scroll down to the bottom of the page until you find the resumptionToken. It will look similar to this: <resumptionToken cursor="50"> oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: </resumptionToken> 4) Copy that resumption token and send a request with it like so: http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords& resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z: 5) The page should (incorrectly) show no records. 6) APPLY PATCH 7) Repeat steps 2, 3, and 4 8) Note that the resumptionToken now uses slashes (e.g. /) instead of colons. Note also that now the second request will show records!!! N.B. This will only happen if Koha has enough records to serve to you. If your Koha has less than 50 records, try lowering the number provided in the "OAI-PMH:MaxCount" system preference. Signed-off-by: Petter Goksoyr Asen <boutrosboutrosboutros@gmail.com> I understand; I can now confirm the behaviour described in the test plan. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes test plan, all tests and QA script. Resumption Token works correctly after applying the patch.
Pushed to master. Thanks, David!
This patch has been pushed to 3.12.x, will be in 3.12.8. Thanks David!