Bug 13568 - OAI Server doesn't handle properly resumption token
Summary: OAI Server doesn't handle properly resumption token
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Frédéric Demians
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 12252
  Show dependency treegraph
 
Reported: 2015-01-13 11:24 UTC by Frédéric Demians
Modified: 2016-06-21 21:37 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 13568 OAI Server doesn't handle properly resumption token (6.38 KB, patch)
2015-01-13 11:28 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 13568 OAI Server doesn't handle properly resumption token (6.45 KB, patch)
2015-02-17 10:58 UTC, sandboxes@biblibre.com
Details | Diff | Splinter Review
Bug 13568 OAI Server doesn't handle properly resumption token (6.51 KB, patch)
2015-02-19 09:47 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Frédéric Demians 2015-01-13 11:24:40 UTC
When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't
return proper resumption token. At the end of a result set, OAI server
generates a resumption token even if there isn't anymore records. Consequently,
OAI harverster will send a new request, based on this invalid resumption,
token. OAI Server responds with an empty resultset, which is considered as an
invalid response by most of the harvesters.
Comment 1 Frédéric Demians 2015-01-13 11:28:41 UTC Comment hidden (obsolete)
Comment 2 Christophe Brocquet 2015-02-17 09:51:04 UTC
I tested the patch for the 13568 with the following sandbox http://pro.test1.biblibre.com/cgi-bin/koha/sandbox.pl
I used the "UNIMARC public library (2) = UNIMARC BM" dataset.

1. First I activated the following system preferences in the staff interface
- OAI-PMH : enabled
- OAI-PMH:AutoUpdateSets : enabled
- OAI-PMH:MaxCount : 500

2. Then I tested with 
- http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListIdentifiers&metadataPrefix=marcxml
- http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListRecords&metadataPrefix=marcxml

3. The last resumtionToken which was generated is "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken.
But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000 bibliographical records.


So I am not sure the patch works well.
Comment 3 Frédéric Demians 2015-02-17 10:26:09 UTC
> 3. The last resumtionToken which was generated is
> "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was
> "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken.
> But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the
> "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000
> bibliographical records.

> So I am not sure the patch works well.

You could see that without the patch, even asking records with "marcxml/4500/1970-01-01/2015-02-17/" resumption token, you will have a resumption token in the response. 

The dataset doesn't contain 19000 records but 4848. You can check with this SQL query: SELECT COUNT(*) FROM biblio.
Comment 4 sandboxes@biblibre.com 2015-02-17 10:58:13 UTC
Patch tested with a sandbox, by Christophe Brocquet <christophe.brocquet@obspm.fr>
Comment 5 sandboxes@biblibre.com 2015-02-17 10:58:37 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2015-02-17 15:54:18 UTC
(In reply to Frédéric Demians from comment #3)
> The dataset doesn't contain 19000 records but 4848. You can check with this
> SQL query: SELECT COUNT(*) FROM biblio.

mysql> select count(*) from biblio;
+----------+
| count(*) |
+----------+
|     4848 |
+----------+
1 row in set (0.00 sec)
Comment 7 Jonathan Druart 2015-02-19 09:47:46 UTC
Created attachment 36018 [details] [review]
Bug 13568 OAI Server doesn't handle properly resumption token

When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't
return proper resumption token. At the end of a result set, OAI server
generates a resumption token even if there isn't anymore records. Consequently,
OAI harverster will send a new request, based on this invalid resumption,
token. OAI Server responds with an empty resultset, which is considered as an
invalid response by most of the harvesters.

TO TEST:

- Find in your DB, a day where a few biblio records have been created. The
  number of created biblios must inferior to OAI-PMH:MaxCount.

- Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server:

  /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09

- At the end of the result, you will see a resumption token which looks like that:

  <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken>

  This is wrong. No resumptiion token should be sent since there isn't anymore
  records to harvest.

- Apply the patch.

- Resend the OAI-PMH request. There is no resumption token at the end of the
  result.

- You could test also with ListIdenfiers verb in place of ListRecord.

Signed-off-by: Christophe Brocquet <christophe.brocquet@obspm.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 8 Tomás Cohen Arazi 2015-02-19 13:12:56 UTC
Patch pushed to master.

Thanks Frederic!
Comment 9 Chris Cormack 2015-02-22 22:59:44 UTC
No string changes pushed to 3.18.x will be in 3.18.4
Comment 10 David Cook 2015-02-23 05:42:01 UTC
Hmm, I haven't had any issues with the resumption tokens... I'll have to take a gander at this sometime...
Comment 11 Frédéric Demians 2015-02-23 07:56:16 UTC
(In reply to David Cook from comment #10)
> Hmm, I haven't had any issues with the resumption tokens... I'll have to
> take a gander at this sometime...

It depends on OAI-PMH harvester: some accepts an empty result set returned to a resumption query. A well known discovery tool, edited by an US-Israeli company, fails to handle an empty resultset, report it, and stop harvesting.
Comment 12 Mason James 2015-02-26 05:11:26 UTC
Possible selection for 3.16.x
Comment 13 Mason James 2015-04-22 12:20:07 UTC
Pushed to 3.16.x, will be in 3.16.10
Comment 14 Fridolin SOMERS 2015-08-31 08:31:25 UTC
This patch will not be in 3.14.x so set as resolved