When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters.
Created attachment 35150 [details] [review] Bug 13568 OAI Server doesn't handle properly resumption token When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. TO TEST: - Find in your DB, a day where a few biblio records have been created. The number of created biblios must inferior to OAI-PMH:MaxCount. - Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09 - At the end of the result, you will see a resumption token which looks like that: <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken> This is wrong. No resumptiion token should be sent since there isn't anymore records to harvest. - Apply the patch. - Resend the OAI-PMH request. There is no resumption token at the end of the result. - You could test also with ListIdenfiers verb in place of ListRecord.
I tested the patch for the 13568 with the following sandbox http://pro.test1.biblibre.com/cgi-bin/koha/sandbox.pl I used the "UNIMARC public library (2) = UNIMARC BM" dataset. 1. First I activated the following system preferences in the staff interface - OAI-PMH : enabled - OAI-PMH:AutoUpdateSets : enabled - OAI-PMH:MaxCount : 500 2. Then I tested with - http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListIdentifiers&metadataPrefix=marcxml - http://catalogue.test1.biblibre.com/cgi-bin/koha/oai.pl/request?verb=ListRecords&metadataPrefix=marcxml 3. The last resumtionToken which was generated is "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken. But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000 bibliographical records. So I am not sure the patch works well.
> 3. The last resumtionToken which was generated is > "marcxml/4500/1970-01-01/2015-02-17/" and the last identifier was > "KOHA-OAI-TEST:5000" : after this record there was no more resumptionToken. > But it is said here http://koha-fr.org/content/bacs-%C3%A0-sable that the > "UNIMARC public library (2) = UNIMARC BM" dataset contains 19000 > bibliographical records. > So I am not sure the patch works well. You could see that without the patch, even asking records with "marcxml/4500/1970-01-01/2015-02-17/" resumption token, you will have a resumption token in the response. The dataset doesn't contain 19000 records but 4848. You can check with this SQL query: SELECT COUNT(*) FROM biblio.
Patch tested with a sandbox, by Christophe Brocquet <christophe.brocquet@obspm.fr>
Created attachment 35962 [details] [review] Bug 13568 OAI Server doesn't handle properly resumption token When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. TO TEST: - Find in your DB, a day where a few biblio records have been created. The number of created biblios must inferior to OAI-PMH:MaxCount. - Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09 - At the end of the result, you will see a resumption token which looks like that: <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken> This is wrong. No resumptiion token should be sent since there isn't anymore records to harvest. - Apply the patch. - Resend the OAI-PMH request. There is no resumption token at the end of the result. - You could test also with ListIdenfiers verb in place of ListRecord. Signed-off-by: Christophe Brocquet <christophe.brocquet@obspm.fr>
(In reply to Frédéric Demians from comment #3) > The dataset doesn't contain 19000 records but 4848. You can check with this > SQL query: SELECT COUNT(*) FROM biblio. mysql> select count(*) from biblio; +----------+ | count(*) | +----------+ | 4848 | +----------+ 1 row in set (0.00 sec)
Created attachment 36018 [details] [review] Bug 13568 OAI Server doesn't handle properly resumption token When responding to ListRecords and ListIdentifiers verbs, OAI server doesn't return proper resumption token. At the end of a result set, OAI server generates a resumption token even if there isn't anymore records. Consequently, OAI harverster will send a new request, based on this invalid resumption, token. OAI Server responds with an empty resultset, which is considered as an invalid response by most of the harvesters. TO TEST: - Find in your DB, a day where a few biblio records have been created. The number of created biblios must inferior to OAI-PMH:MaxCount. - Let say this day is 2014-01-09. Send an OAI-PMH request to Koha OAI Server: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=marcxml&from=2014-01-09&until=2014-01-09 - At the end of the result, you will see a resumption token which looks like that: <resumptionToken cursor="47">marcxml/47/2014-01-09/2014-01-09/</resumptionToken> This is wrong. No resumptiion token should be sent since there isn't anymore records to harvest. - Apply the patch. - Resend the OAI-PMH request. There is no resumption token at the end of the result. - You could test also with ListIdenfiers verb in place of ListRecord. Signed-off-by: Christophe Brocquet <christophe.brocquet@obspm.fr> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Patch pushed to master. Thanks Frederic!
No string changes pushed to 3.18.x will be in 3.18.4
Hmm, I haven't had any issues with the resumption tokens... I'll have to take a gander at this sometime...
(In reply to David Cook from comment #10) > Hmm, I haven't had any issues with the resumption tokens... I'll have to > take a gander at this sometime... It depends on OAI-PMH harvester: some accepts an empty result set returned to a resumption query. A well known discovery tool, edited by an US-Israeli company, fails to handle an empty resultset, report it, and stop harvesting.
Possible selection for 3.16.x
Pushed to 3.16.x, will be in 3.16.10
This patch will not be in 3.14.x so set as resolved