The OAI-PMH provider may end up in an eternal loop when processing deleted records due to the select statement missing sort (and limit). If records are not returned from the database in biblionumber order naturally, this omission may lead to the same records to be processed with each subsequent request, or records to be skipped. Fix coming up.
Note: this can only happen when include_items is enabled in oai.conf, which contributes to it lurking around as long as it has.
Created attachment 130961 [details] [review] Bug 30143: OAI-PMH: Fix SQL query used to fetch deleted records The query was missing the "ORDER BY" and "LIMIT" clauses, which could make retrieve a wrong set of records.
I can't provide steps to reproduce since my database always returns the records in biblionumber order when querying without a specific ORDER BY clause. This issue was, however, encountered in a production system. The mistake originates from refactoring done in bug 29135.
Depends on bug 29135. The fix is correct, however maybe it would make to sense to move out the common SQL above the if-else condition like it was before bug 29135, this way we wouldn't accidentally forget to change this SQL query in both places in the future. Not a blocker for this, gonna add my sign-off.
Created attachment 133703 [details] [review] Bug 30143: OAI-PMH: Fix SQL query used to fetch deleted records The query was missing the "ORDER BY" and "LIMIT" clauses, which could make retrieve a wrong set of records. Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi>
Created attachment 134352 [details] [review] Bug 30143: OAI-PMH: Fix SQL query used to fetch deleted records The query was missing the "ORDER BY" and "LIMIT" clauses, which could make retrieve a wrong set of records. Signed-off-by: Joonas Kylmälä <joonas.kylmala@iki.fi> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Pushed to master for 22.05, thanks to everybody involved [U+1F984]
Note: it has been backported to 21.11.x
Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed.
Architectural fix, no documentation change required.