Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time
Summary: OAI-PMH Resumption Tokens Do Not Handle Time
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: David Cook
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-09-30 05:29 UTC by David Cook
Modified: 2015-06-04 23:23 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time (2.51 KB, patch)
2013-09-30 05:54 UTC, David Cook
Details | Diff | Splinter Review
Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time (2.65 KB, patch)
2013-10-21 23:57 UTC, Petter Goksøyr Åsen
Details | Diff | Splinter Review
[PASSED QA] Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time (2.82 KB, patch)
2013-10-27 15:44 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2013-09-30 05:29:45 UTC
Currently, it is possible to retrieve 50 or so records from Koha via OAI-PMH, when using times in your 'from' and 'until' arguments. Here is an example that will return a list of records (if your OAI server syspref is turned on).

http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z

However, if you need to retrieve the rest of the results via a resumption token, Koha won't return your results.

http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z:

This is because Koha joins and splits the resumption token using colons. Obviously, since the timestamps use colons, the string isn't going to be split correctly. 

I propose that we change the separator colons to slashes. This is the method that DSpace uses when serving records via OAI-PMH. You can see an example here:

http://papyrus.bib.umontreal.ca/dspace-oai/request?verb=ListRecords&metadataPrefix=oai_dc

--

Note: Koha doesn't handle times 100% correctly in master either. That's why I opened bug 10824.
Comment 1 David Cook 2013-09-30 05:54:46 UTC Comment hidden (obsolete)
Comment 2 I'm just a bot 2013-10-01 18:40:38 UTC
Patch applied cleanly, go forth and signoff
Comment 3 Petter Goksøyr Åsen 2013-10-21 19:08:50 UTC
Strange, the resumptionToken request works for me, without applying the patch.

It also works after I apply the patch of course.
Comment 4 David Cook 2013-10-21 22:44:23 UTC
(In reply to Petter Goksøyr Åsen from comment #3)
> Strange, the resumptionToken request works for me, without applying the
> patch.
> 
> It also works after I apply the patch of course.

Thanks for that Petter. I should've written my test plan better.

In fact, it only looks like it's working.

If you insert some debugging code, you will find that the following token:

oai_dc:50:2012-09-05T13:44:33Z:2013-09-05T13:44:33Z:

is being broken into the following:

metadataPrefix = oai_dc
offset = 50
from = 2012-09-05T13
until = 44
set = 33Z

and the rest of the token will be thrown away.

--

If you re-run the test, take a look at the first result you receive in the ListRecords response. Change your "until" date to before the datestamp of that first result.

You will find that record still appears when it shouldn't!

--

Also, the resumption token at the bottom will be similar to the following:

oai_dc:66:2012-09-05T13:44:33Z

This looks like a resumption token with a fully formed From date. 

However, it's a token built up from those split fields I identified before. It thinks 44 is the until datestamp and 33Z is the set. 

If you try to use that resumptionToken, you will get no results.

--

Please try it again in light of these notes. I'm sure that you'll find I'm correct :)
Comment 5 Petter Goksøyr Åsen 2013-10-21 23:57:22 UTC Comment hidden (obsolete)
Comment 6 Katrin Fischer 2013-10-27 15:44:49 UTC
Created attachment 22450 [details] [review]
[PASSED QA] Bug 10974 - OAI-PMH Resumption Tokens Do Not Handle Time

This patch changes the value separator in OAI-PMH resumption tokens
from colons to slashes, so that the token string isn't split incorrectly
when a time is included.

TEST PLAN:

1) Turn on the OAI-PMH server syspref in Koha
2) Send a ListRecords request using 'from' and 'until' arguments that
include times (Best to use very far apart times so that you retrieve
more than 50 records which will likely be the trigger for a resumptionToken).
Here is an example:

http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&
metadataPrefix=oai_dc&from=2012-09-05T13:44:33Z&until=2014-09-05T13:44:33Z

N.B. Replace KOHAINSTANCE with the URL of your Koha instance.

3) Scroll down to the bottom of the page until you find the resumptionToken.
It will look similar to this:

<resumptionToken cursor="50">
oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z:
</resumptionToken>

4) Copy that resumption token and send a request with it like so:

http://KOHAINSTANCE/cgi-bin/koha/oai.pl?verb=ListRecords&
resumptionToken=oai_dc:50:2012-09-05T13:44:33Z:2014-09-05T13:44:33Z:

5) The page should (incorrectly) show no records.

6) APPLY PATCH

7) Repeat steps 2, 3, and 4

8) Note that the resumptionToken now uses slashes (e.g. /) instead of
colons.

Note also that now the second request will show records!!!

N.B. This will only happen if Koha has enough records to serve to you.
If your Koha has less than 50 records, try lowering the number provided
in the "OAI-PMH:MaxCount" system preference.

Signed-off-by: Petter Goksoyr Asen <boutrosboutrosboutros@gmail.com>

I understand; I can now confirm the behaviour described in the test plan.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes test plan, all tests and QA script.
Resumption Token works correctly after applying the patch.
Comment 7 Galen Charlton 2013-10-31 18:03:17 UTC
Pushed to master.  Thanks, David!
Comment 8 Tomás Cohen Arazi 2013-12-05 18:27:16 UTC
This patch has been pushed to 3.12.x, will be in 3.12.8.

Thanks David!