Bug 3206 - OAI-PMH repository deleted record support
Summary: OAI-PMH repository deleted record support
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Web services (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Matthias Meusburger
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 8149 12252
  Show dependency treegraph
 
Reported: 2009-05-08 01:21 UTC by Chris Cormack
Modified: 2016-12-05 21:34 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 3206: OAI repository deleted records support. (10.14 KB, patch)
2015-06-02 14:56 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 3206: OAI repository deleted records support. (11.24 KB, patch)
2015-06-03 07:00 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 3206: OAI repository deleted records support. (11.38 KB, patch)
2015-06-03 08:54 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 3206: OAI repository deleted records support. (11.38 KB, patch)
2015-06-03 10:03 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 3206 [Follow-up]: OAI repository deleted records support. (5.48 KB, patch)
2015-06-08 13:34 UTC, Matthias Meusburger
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 3206 [Follow-up]: OAI repository deleted records support. (5.67 KB, patch)
2015-06-19 23:21 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 3206: OAI repository deleted records support. (11.44 KB, patch)
2015-06-22 11:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 3206 [Follow-up]: OAI repository deleted records support. (5.72 KB, patch)
2015-06-22 11:34 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 3206: OAI repository deleted records support. (11.51 KB, patch)
2015-06-22 14:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 3206 [Follow-up]: OAI repository deleted records support. (5.79 KB, patch)
2015-06-22 14:08 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 3206: (RM followup) DBIx updates (3.98 KB, patch)
2015-07-08 18:07 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 3206: (QA followup) missing comma on sysprefs.sql (1.81 KB, patch)
2015-07-08 21:18 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2010-05-21 01:07:40 UTC


---- Reported by eosisek@library.rochester.edu 2009-05-08 13:21:00 ----

Koha's OAI repository reports that it does not track deleted records. (http://gmc.dev.kohalibrary.com/cgi-bin/koha/oai.pl?verb=Identify)  We would like Koha's OAI repository to have "persistent" support for deleted records, as defined by the OAI protocol. (http://www.openarchives.org/OAI/openarchivesprotocol.html#DeletedRecords)  Without deleted record support, tools which use OAI will have to do a full harvest of all the records in Koha in order to remain in sync with the data in Koha.  Deleted record support would enable these tools to mirror the data in Koha using incremental harvests, which are much more efficient with large data sets.



--- Bug imported by chris@bigballofwax.co.nz 2010-05-21 01:07 UTC  ---

This bug was previously known as _bug_ 3206 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=3206

Actual time not defined. Setting to 0.0
The original reporter of this bug does not have
   an account here. Reassigning to the person who moved
   it here: chris@bigballofwax.co.nz.
   Previous reporter was eosisek@library.rochester.edu.

Comment 1 Frédéric Demians 2012-05-25 05:12:48 UTC
This is an easy enhancement. 'deletedbiblio' table contains all the
information required to add deleted records support to Koha OAI Server.
Currently, selective harvesting replies with added/modified biblio
records. We just have to request 'deletedbiblio' table to send back also
deleted records.
Comment 2 David Cook 2013-08-19 06:57:54 UTC
(In reply to Frédéric Demians from comment #1)
> This is an easy enhancement. 'deletedbiblio' table contains all the
> information required to add deleted records support to Koha OAI Server.
> Currently, selective harvesting replies with added/modified biblio
> records. We just have to request 'deletedbiblio' table to send back also
> deleted records.

Sounds right to me. We'd also need to add in a status element saying "deleted". I suppose declaring the repo "persistent" vs "transient" would depend on if the library ever deletes their deletedbiblio table.
Comment 3 David Cook 2013-08-19 07:10:55 UTC
(In reply to David Cook from comment #2)
> (In reply to Frédéric Demians from comment #1)
> > This is an easy enhancement. 'deletedbiblio' table contains all the
> > information required to add deleted records support to Koha OAI Server.
> > Currently, selective harvesting replies with added/modified biblio
> > records. We just have to request 'deletedbiblio' table to send back also
> > deleted records.
> 
> Sounds right to me. We'd also need to add in a status element saying
> "deleted". I suppose declaring the repo "persistent" vs "transient" would
> depend on if the library ever deletes their deletedbiblio table.

My mistake...the header element should actually have an attribute with a value of deleted, based on the examples from http://www.openarchives.org/OAI/openarchivesprotocol.html
Comment 4 David Cook 2013-09-30 01:54:47 UTC
Something to check into would be how biblios are deleted when a batch import is reverted...we'll need to make sure that the bibs are logged in the deletedbiblio table and not just nuked.
Comment 5 Frédéric Demians 2013-10-02 06:12:53 UTC
> Something to check into would be how biblios are deleted when a batch
> import is reverted...we'll need to make sure that the bibs are logged
> in the deletedbiblio table and not just nuked.

Yes, there may be some intricacies here, and some new added by OAI Sets.
What will happen if a SET definition is modified in Koha? As it is,
harvesters must be notified to re-harvest Koha repository from scratch.
Comment 6 Matthias Meusburger 2015-06-02 14:56:10 UTC Comment hidden (obsolete)
Comment 7 Frédéric Demians 2015-06-02 15:33:46 UTC
Adding a new syspref, you're supposed to add it also to sysprefs.sql file. See:

http://wiki.koha-community.org/wiki/System_Preferences


I don't understand this in your patch:

diff --git a/installer/data/mysql/kohastructure.sql b/installer/data/mysql/kohastructure.sql
index c41710c..5fd5612 100644
--- a/installer/data/mysql/kohastructure.sql
+++ b/installer/data/mysql/kohastructure.sql
@@ -1609,7 +1609,6 @@ CREATE TABLE `oai_sets_biblios` (
   `biblionumber` int(11) NOT NULL,
   `set_id` int(11) NOT NULL,
   PRIMARY KEY (`biblionumber`, `set_id`),
-  CONSTRAINT `oai_sets_biblios_ibfk_1` FOREIGN KEY (`biblionumber`) REFERENCES `biblio` (`biblionumber`) ON DELETE CASCADE ON UPDATE CASCADE,
   CONSTRAINT `oai_sets_biblios_ibfk_2` FOREIGN KEY (`set_id`) REFERENCES `oai_sets` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
 ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Comment 8 Matthias Meusburger 2015-06-03 07:00:57 UTC Comment hidden (obsolete)
Comment 9 Matthias Meusburger 2015-06-03 07:03:55 UTC
(In reply to Frédéric Demians from comment #7)
> Adding a new syspref, you're supposed to add it also to sysprefs.sql file.

Added ! 

> 
> I don't understand this in your patch:
> 
> diff --git a/installer/data/mysql/kohastructure.sql
> b/installer/data/mysql/kohastructure.sql
> index c41710c..5fd5612 100644
> --- a/installer/data/mysql/kohastructure.sql
> +++ b/installer/data/mysql/kohastructure.sql
> @@ -1609,7 +1609,6 @@ CREATE TABLE `oai_sets_biblios` (
>    `biblionumber` int(11) NOT NULL,
>    `set_id` int(11) NOT NULL,
>    PRIMARY KEY (`biblionumber`, `set_id`),
> -  CONSTRAINT `oai_sets_biblios_ibfk_1` FOREIGN KEY (`biblionumber`)
> REFERENCES `biblio` (`biblionumber`) ON DELETE CASCADE ON UPDATE CASCADE,
>    CONSTRAINT `oai_sets_biblios_ibfk_2` FOREIGN KEY (`set_id`) REFERENCES
> `oai_sets` (`id`) ON DELETE CASCADE ON UPDATE CASCADE
>  ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

This is required to be able to display deleted records in a set.
With the constraint, when a record is deleted, its biblionumber is automatically deleted from the oai_sets_biblio, and thus removed from the set.
Comment 10 Frédéric Demians 2015-06-03 08:54:28 UTC Comment hidden (obsolete)
Comment 11 Frédéric Demians 2015-06-03 10:03:11 UTC Comment hidden (obsolete)
Comment 12 Jonathan Druart 2015-06-03 12:20:31 UTC
Comment on attachment 39797 [details] [review]
Bug 3206: OAI repository deleted records support.

Review of attachment 39797 [details] [review]:
-----------------------------------------------------------------

::: installer/data/mysql/sysprefs.sql
@@ +479,5 @@
>  ('XSLTDetailsDisplay','default','','Enable XSL stylesheet control over details page display on intranet','Free'),
>  ('XSLTResultsDisplay','default','','Enable XSL stylesheet control over results page display on intranet','Free'),
>  ('z3950AuthorAuthFields','701,702,700',NULL,'Define the MARC biblio fields for Personal Name Authorities to fill biblio.author','free'),
> +('z3950NormalizeAuthor','0','','If ON, Personal Name Authorities will replace authors in biblio.author','YesNo'),
> +('OAI-PMH:DeletedRecord','persistent','Koha\'s deletedbiblio table will never be deleted (persistent) or might be deleted (transient)','transient|persistent','Choice')

Insert the pref at the right place, this file must be kept ordered.

::: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref
@@ +31,5 @@
> +            - Koha's deletedbiblio table
> +            - pref: "OAI-PMH:DeletedRecord"
> +              choices:
> +                  persistent: will never be emptied or truncated (persistent)
> +                  transient: might be emptied or truncated at some point (transient)

Does it make sense to have the "no" option?

::: opac/oai.pl
@@ +242,5 @@
>  
> +package C4::OAI::DeletedRecord;
> +
> +use strict;
> +use warnings;

Prefer Modern::Perl;

@@ +303,5 @@
> +        FROM deletedbiblio
> +        WHERE biblionumber=? " );
> +      $sth->execute( $biblionumber );
> +
> +      unless ( ($marcxml, $timestamp) = $sth->fetchrow ) {

Prefer $dbh->selectrow_*

@@ +358,4 @@
>      }
>      my $max = $repository->{koha_max_count};
>      my $sql = "
> +        (SELECT biblioitems.biblionumber, timestamp

Why do you remove the table name?
Comment 13 Jonathan Druart 2015-06-03 12:24:15 UTC
Should not we update the misc/migration_tools/build_oai_sets.pl script to take into account the deletedbiblio table?
Comment 14 Matthias Meusburger 2015-06-03 12:31:30 UTC
(In reply to Jonathan Druart from comment #13)
> Should not we update the misc/migration_tools/build_oai_sets.pl script to
> take into account the deletedbiblio table?

My guess is that we can't (not straightforwardly at least). 

OAI sets are built from rules based on MARC informations. As marcxml is not kept in the deletedbiblio table, we can't use those rules to determine which deleted records belong to a given set.

However, we could decide to keep the marcxml in the deletedbiblio table, or to add a field to the deletebiblio table with the oai set id the record used to belong to before its deletion.
Comment 15 Gaetan Boisson 2015-06-03 14:00:51 UTC
Jonathan, i think the "no" option (and the whole syspref) is there to stick to the OAI standard:
http://www.openarchives.org/OAI/openarchivesprotocol.html#DeletedRecords

My understanding is that the transient, persistent or no status, is more a result of how you administer your database than one of the choice you made in this syspref. I'd say it's here so that the server communicates the right value to harvesters. Default value should be persistent, since the deletedbiblio table can only be emptied manually if the administrator does so. So transient would be "i do that from time to time" and 'no' would be "i never ever let anything go in this table".
Comment 16 Katrin Fischer 2015-06-04 13:36:37 UTC
Hm, I think initially including the deleted records when creating a set seems to go to far - you might probably not want those at this point. But once the set is created you want to inform about records removed from it. So changing the cronjob might not be necessary?
Comment 17 Matthias Meusburger 2015-06-08 13:34:12 UTC Comment hidden (obsolete)
Comment 18 Jonathan Druart 2015-06-16 11:40:25 UTC
I would like to get a signoff on the second patch.
Comment 19 Bernardo Gonzalez Kriegel 2015-06-19 23:21:03 UTC Comment hidden (obsolete)
Comment 20 Jonathan Druart 2015-06-22 11:34:48 UTC Comment hidden (obsolete)
Comment 21 Jonathan Druart 2015-06-22 11:34:56 UTC Comment hidden (obsolete)
Comment 22 Jonathan Druart 2015-06-22 11:36:13 UTC
(In reply to Katrin Fischer from comment #16)
> Hm, I think initially including the deleted records when creating a set
> seems to go to far - you might probably not want those at this point. But
> once the set is created you want to inform about records removed from it. So
> changing the cronjob might not be necessary?

An answer is needed here.
Comment 23 Gaetan Boisson 2015-06-22 13:43:46 UTC
I agree with Katrin, initially including deleted records seems a bit overkill to me.
Comment 24 Jonathan Druart 2015-06-22 14:08:45 UTC
Created attachment 40478 [details] [review]
Bug 3206: OAI repository deleted records support.

This patch allows Koha OAI repository to support deleted records.

The OAI-PMH:DeletedRecord syspref is introduced and can be set to:

- persistent (in case Koha's deletedbiblio table will never be emptied
  or truncated)
- transient (in case Koha's deletedbiblio table might be emptied or
  truncated at some point)

Test plan:

- After applying the patch, test that:
   - Deleted records appear in ListRecords and ListIdentifiers requests.
   - Filter parameters (from, until, set and resumptionToken) still work
     and are applied to ListRecords and ListIdentifiers requests.
   - Identify request shows if the repository is considered persistent
     or transient, according to the OAI-PMH:DeletedRecord syspref.
   - Deleted records that used to belong to a set are still displayed in
     those sets and marked as deleted.
   - GetRecord requests work on deleted records, which are marked as deleted.

Requests examples:
/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc
/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z
/cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1
/cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc
/cgi-bin/koha/oai.pl?verb=Identify

Signed-off-by: Frederic Demians <f.demians@tamil.fr>
  It works in all situations described in the test plan. Great addition.
  Thanks.

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 25 Jonathan Druart 2015-06-22 14:08:52 UTC
Created attachment 40479 [details] [review]
Bug 3206 [Follow-up]: OAI repository deleted records support.

- Fix QA.

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
New pref in right order, new option 'no' on syspref, other
fixes following comment #12
All seems to work
No errors

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Comment 26 Tomás Cohen Arazi 2015-07-08 18:07:49 UTC
Created attachment 40846 [details] [review]
Bug 3206: (RM followup) DBIx updates

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 27 Tomás Cohen Arazi 2015-07-08 18:20:34 UTC
Enahcement pushed to master.

Good job Matthias!
Comment 28 Tomás Cohen Arazi 2015-07-08 21:18:45 UTC
Created attachment 40849 [details] [review]
Bug 3206: (QA followup) missing comma on sysprefs.sql

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>