Bug 13568

Summary: OAI Server doesn't handle properly resumption token
Product: Koha Reporter: Frédéric Demians <f.demians>
Component: Web servicesAssignee: Frédéric Demians <f.demians>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: chris, christophe.brocquet, dcook, fridolin.somers, jonathan.druart, mtj, sandboxes, tomascohen
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on:    
Bug Blocks: 12252    
Attachments: Bug 13568 OAI Server doesn't handle properly resumption token
Bug 13568 OAI Server doesn't handle properly resumption token
Bug 13568 OAI Server doesn't handle properly resumption token

Description Frédéric Demians 2015-01-13 11:24:40 UTC
When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't
return proper resumption token. At the end of a result set, OAI server
generates a resumption token even if there isn't anymore records. Consequently,
OAI harverster will send a new request, based on this invalid resumption,
token. OAI Server responds with an empty resultset, which is considered as an
invalid response by most of the harvesters.
Comment 1 Frédéric Demians 2015-01-13 11:28:41 UTC Comment hidden (obsolete)
Comment 2 Christophe Brocquet 2015-02-17 09:51:04 UTC
I tested the patch for the 13568 with the following sandbox http://pro.test1.biblibre.com/cgi-bin/koha/sandbox.pl
I used the "UNIMARC public library (2) = UNIMARC BM" dataset.

1. First I activated the following system preferences in the staff interface
- OAI-PMH : enabled
- OAI-PMH:AutoUpdateSets : enabled
- OAI-PMH:MaxCount : 500

2. Then I tested with 
- http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListIdentifiers&metadataPrefix=marcxml
- http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListRecords&metadataPrefix=marcxml

3. The last resumtionToken which was generated is "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken.
But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000 bibliographical records.


So I am not sure the patch works well.
Comment 3 Frédéric Demians 2015-02-17 10:26:09 UTC
> 3. The last resumtionToken which was generated is
> "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was
> "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken.
> But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the
> "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000
> bibliographical records.

> So I am not sure the patch works well.

You could see that without the patch, even asking records with "marcxml/4500/1970-01-01/2015-02-17/" resumption token, you will have a resumption token in the response. 

The dataset doesn't contain 19000 records but 4848. You can check with this SQL query: SELECT COUNT(*) FROM biblio.
Comment 4 Biblibre Sandboxes 2015-02-17 10:58:13 UTC
Patch tested with a sandbox, by Christophe Brocquet <christophe.brocquet@obspm.fr>
Comment 5 Biblibre Sandboxes 2015-02-17 10:58:37 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2015-02-17 15:54:18 UTC
(In reply to Frédéric Demians from comment #3)
> The dataset doesn't contain 19000 records but 4848. You can check with this
> SQL query: SELECT COUNT(*) FROM biblio.

mysql> select count(*) from biblio;
+----------+
| count(*) |
+----------+
|     4848 |
+----------+
1 row in set (0.00 sec)
Comment 7 Jonathan Druart 2015-02-19 09:47:46 UTC
Created attachment 36018 [details] [review]
Bug 13568 OAI Server doesn't handle properly resumption token

When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't
return proper resumption token. At the end of a result set, OAI server
generates a resumption token even if there isn't anymore records. Consequently,
OAI harverster will send a new request, based on this invalid resumption,
token. OAI Server responds with an empty resultset, which is considered as an
invalid response by most of the harvesters.

TO TEST:

- Find in your DB, a day where a few biblio records have been created. The
  number of created biblios must inferior to OAI-PMH:MaxCount.

- Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server:

  /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09

- At the end of the result, you will see a resumption token which looks like that:

  <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken>

  This is wrong. No resumptiion token should be sent since there isn't anymore
  records to harvest.

- Apply the patch.

- Resend the OAI-PMH request. There is no resumption token at the end of the
  result.

- You could test also with ListIdenfiers verb in place of ListRecord.

Signed-off-by: Christophe Brocquet <christophe.brocquet@obspm.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 8 Tomás Cohen Arazi 2015-02-19 13:12:56 UTC
Patch pushed to master.

Thanks Frederic!
Comment 9 Chris Cormack 2015-02-22 22:59:44 UTC
No string changes pushed to 3.18.x will be in 3.18.4
Comment 10 David Cook 2015-02-23 05:42:01 UTC
Hmm, I haven't had any issues with the resumption tokens... I'll have to take a gander at this sometime...
Comment 11 Frédéric Demians 2015-02-23 07:56:16 UTC
(In reply to David Cook from comment #10)
> Hmm, I haven't had any issues with the resumption tokens... I'll have to
> take a gander at this sometime...

It depends on OAI-PMH harvester: some accepts an empty result set returned to a resumption query. A well known discovery tool, edited by an US-Israeli company, fails to handle an empty resultset, report it, and stop harvesting.
Comment 12 Mason James 2015-02-26 05:11:26 UTC
Possible selection for 3.16.x
Comment 13 Mason James 2015-04-22 12:20:07 UTC
Pushed to 3.16.x, will be in 3.16.10
Comment 14 Fridolin Somers 2015-08-31 08:31:25 UTC
This patch will not be in 3.14.x so set as resolved