---- Reported by eosisek@library.rochester.edu 2009-05-08 13:21:00 ---- Koha's OAI repository reports that it does not track deleted records. (http://gmc.dev.kohalibrary.com/cgi-bin/koha/oai.pl?verb=Identify) We would like Koha's OAI repository to have "persistent" support for deleted records, as defined by the OAI protocol. (http://www.openarchives.org/OAI/openarchivesprotocol.html#DeletedRecords) Without deleted record support, tools which use OAI will have to do a full harvest of all the records in Koha in order to remain in sync with the data in Koha. Deleted record support would enable these tools to mirror the data in Koha using incremental harvests, which are much more efficient with large data sets. --- Bug imported by chris@bigballofwax.co.nz 2010-05-21 01:07 UTC --- This bug was previously known as _bug_ 3206 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=3206 Actual time not defined. Setting to 0.0 The original reporter of this bug does not have an account here. Reassigning to the person who moved it here: chris@bigballofwax.co.nz. Previous reporter was eosisek@library.rochester.edu.
This is an easy enhancement. 'deletedbiblio' table contains all the information required to add deleted records support to Koha OAI Server. Currently, selective harvesting replies with added/modified biblio records. We just have to request 'deletedbiblio' table to send back also deleted records.
(In reply to Frédéric Demians from comment #1) > This is an easy enhancement. 'deletedbiblio' table contains all the > information required to add deleted records support to Koha OAI Server. > Currently, selective harvesting replies with added/modified biblio > records. We just have to request 'deletedbiblio' table to send back also > deleted records. Sounds right to me. We'd also need to add in a status element saying "deleted". I suppose declaring the repo "persistent" vs "transient" would depend on if the library ever deletes their deletedbiblio table.
(In reply to David Cook from comment #2) > (In reply to Frédéric Demians from comment #1) > > This is an easy enhancement. 'deletedbiblio' table contains all the > > information required to add deleted records support to Koha OAI Server. > > Currently, selective harvesting replies with added/modified biblio > > records. We just have to request 'deletedbiblio' table to send back also > > deleted records. > > Sounds right to me. We'd also need to add in a status element saying > "deleted". I suppose declaring the repo "persistent" vs "transient" would > depend on if the library ever deletes their deletedbiblio table. My mistake...the header element should actually have an attribute with a value of deleted, based on the examples from http://www.openarchives.org/OAI/openarchivesprotocol.html
Something to check into would be how biblios are deleted when a batch import is reverted...we'll need to make sure that the bibs are logged in the deletedbiblio table and not just nuked.
> Something to check into would be how biblios are deleted when a batch > import is reverted...we'll need to make sure that the bibs are logged > in the deletedbiblio table and not just nuked. Yes, there may be some intricacies here, and some new added by OAI Sets. What will happen if a SET definition is modified in Koha? As it is, harvesters must be notified to re-harvest Koha repository from scratch.
Created attachment 39771 [details] [review] Bug 3206: OAI repository deleted records support. This patch allows Koha OAI repository to support deleted records. The OAI-PMH:DeletedRecord syspref is introduced and can be set to: - persistent (in case Koha's deletedbiblio table will never be emptied or truncated) - transient (in case Koha's deletedbiblio table might be emptied or truncated at some point) Test plan: - After applying the patch, test that: - Deleted records appear in ListRecords and ListIdentifiers requests. - Filter parameters (from, until, set and resumptionToken) still work and are applied to ListRecords and ListIdentifiers requests. - Identify request shows if the repository is considered persistent or transient, according to the OAI-PMH:DeletedRecord syspref. - Deleted records that used to belong to a set are still displayed in those sets and marked as deleted. - GetRecord requests work on deleted records, which are marked as deleted. Requests examples: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1 /cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=Identify
Adding a new syspref, you're supposed to add it also to sysprefs.sql file. See: http://wiki.koha-community.org/wiki/System_Preferences I don't understand this in your patch: diff --git a/installer/data/mysql/kohastructure.sql b/installer/data/mysql/kohastructure.sql index c41710c..5fd5612 100644 --- a/installer/data/mysql/kohastructure.sql +++ b/installer/data/mysql/kohastructure.sql @@ -1609,7 +1609,6 @@ CREATE TABLE `oai_sets_biblios` ( `biblionumber` int(11) NOT NULL, `set_id` int(11) NOT NULL, PRIMARY KEY (`biblionumber`, `set_id`), - CONSTRAINT `oai_sets_biblios_ibfk_1` FOREIGN KEY (`biblionumber`) REFERENCES `biblio` (`biblionumber`) ON DELETE CASCADE ON UPDATE CASCADE, CONSTRAINT `oai_sets_biblios_ibfk_2` FOREIGN KEY (`set_id`) REFERENCES `oai_sets` (`id`) ON DELETE CASCADE ON UPDATE CASCADE ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;
Created attachment 39794 [details] [review] Bug 3206: OAI repository deleted records support. This patch allows Koha OAI repository to support deleted records. The OAI-PMH:DeletedRecord syspref is introduced and can be set to: - persistent (in case Koha's deletedbiblio table will never be emptied or truncated) - transient (in case Koha's deletedbiblio table might be emptied or truncated at some point) Test plan: - After applying the patch, test that: - Deleted records appear in ListRecords and ListIdentifiers requests. - Filter parameters (from, until, set and resumptionToken) still work and are applied to ListRecords and ListIdentifiers requests. - Identify request shows if the repository is considered persistent or transient, according to the OAI-PMH:DeletedRecord syspref. - Deleted records that used to belong to a set are still displayed in those sets and marked as deleted. - GetRecord requests work on deleted records, which are marked as deleted. Requests examples: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1 /cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=Identify
(In reply to Frédéric Demians from comment #7) > Adding a new syspref, you're supposed to add it also to sysprefs.sql file. Added ! > > I don't understand this in your patch: > > diff --git a/installer/data/mysql/kohastructure.sql > b/installer/data/mysql/kohastructure.sql > index c41710c..5fd5612 100644 > --- a/installer/data/mysql/kohastructure.sql > +++ b/installer/data/mysql/kohastructure.sql > @@ -1609,7 +1609,6 @@ CREATE TABLE `oai_sets_biblios` ( > `biblionumber` int(11) NOT NULL, > `set_id` int(11) NOT NULL, > PRIMARY KEY (`biblionumber`, `set_id`), > - CONSTRAINT `oai_sets_biblios_ibfk_1` FOREIGN KEY (`biblionumber`) > REFERENCES `biblio` (`biblionumber`) ON DELETE CASCADE ON UPDATE CASCADE, > CONSTRAINT `oai_sets_biblios_ibfk_2` FOREIGN KEY (`set_id`) REFERENCES > `oai_sets` (`id`) ON DELETE CASCADE ON UPDATE CASCADE > ) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci; This is required to be able to display deleted records in a set. With the constraint, when a record is deleted, its biblionumber is automatically deleted from the oai_sets_biblio, and thus removed from the set.
Created attachment 39795 [details] [review] Bug 3206: OAI repository deleted records support. This patch allows Koha OAI repository to support deleted records. The OAI-PMH:DeletedRecord syspref is introduced and can be set to: - persistent (in case Koha's deletedbiblio table will never be emptied or truncated) - transient (in case Koha's deletedbiblio table might be emptied or truncated at some point) Test plan: - After applying the patch, test that: - Deleted records appear in ListRecords and ListIdentifiers requests. - Filter parameters (from, until, set and resumptionToken) still work and are applied to ListRecords and ListIdentifiers requests. - Identify request shows if the repository is considered persistent or transient, according to the OAI-PMH:DeletedRecord syspref. - Deleted records that used to belong to a set are still displayed in those sets and marked as deleted. - GetRecord requests work on deleted records, which are marked as deleted. Requests examples: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1 /cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=Identify Signed-off-by: Frederic Demians <f.demians@tamil.fr> It works in all situations described in the test plan. Great addition. Thanks.
Created attachment 39797 [details] [review] Bug 3206: OAI repository deleted records support. This patch allows Koha OAI repository to support deleted records. The OAI-PMH:DeletedRecord syspref is introduced and can be set to: - persistent (in case Koha's deletedbiblio table will never be emptied or truncated) - transient (in case Koha's deletedbiblio table might be emptied or truncated at some point) Test plan: - After applying the patch, test that: - Deleted records appear in ListRecords and ListIdentifiers requests. - Filter parameters (from, until, set and resumptionToken) still work and are applied to ListRecords and ListIdentifiers requests. - Identify request shows if the repository is considered persistent or transient, according to the OAI-PMH:DeletedRecord syspref. - Deleted records that used to belong to a set are still displayed in those sets and marked as deleted. - GetRecord requests work on deleted records, which are marked as deleted. Requests examples: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1 /cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=Identify Signed-off-by: Frederic Demians <f.demians@tamil.fr> It works in all situations described in the test plan. Great addition. Thanks.
Comment on attachment 39797 [details] [review] Bug 3206: OAI repository deleted records support. Review of attachment 39797 [details] [review]: ----------------------------------------------------------------- ::: installer/data/mysql/sysprefs.sql @@ +479,5 @@ > ('XSLTDetailsDisplay','default','','Enable XSL stylesheet control over details page display on intranet','Free'), > ('XSLTResultsDisplay','default','','Enable XSL stylesheet control over results page display on intranet','Free'), > ('z3950AuthorAuthFields','701,702,700',NULL,'Define the MARC biblio fields for Personal Name Authorities to fill biblio.author','free'), > +('z3950NormalizeAuthor','0','','If ON, Personal Name Authorities will replace authors in biblio.author','YesNo'), > +('OAI-PMH:DeletedRecord','persistent','Koha\'s deletedbiblio table will never be deleted (persistent) or might be deleted (transient)','transient|persistent','Choice') Insert the pref at the right place, this file must be kept ordered. ::: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/web_services.pref @@ +31,5 @@ > + - Koha's deletedbiblio table > + - pref: "OAI-PMH:DeletedRecord" > + choices: > + persistent: will never be emptied or truncated (persistent) > + transient: might be emptied or truncated at some point (transient) Does it make sense to have the "no" option? ::: opac/oai.pl @@ +242,5 @@ > > +package C4::OAI::DeletedRecord; > + > +use strict; > +use warnings; Prefer Modern::Perl; @@ +303,5 @@ > + FROM deletedbiblio > + WHERE biblionumber=? " ); > + $sth->execute( $biblionumber ); > + > + unless ( ($marcxml, $timestamp) = $sth->fetchrow ) { Prefer $dbh->selectrow_* @@ +358,4 @@ > } > my $max = $repository->{koha_max_count}; > my $sql = " > + (SELECT biblioitems.biblionumber, timestamp Why do you remove the table name?
Should not we update the misc/migration_tools/build_oai_sets.pl script to take into account the deletedbiblio table?
(In reply to Jonathan Druart from comment #13) > Should not we update the misc/migration_tools/build_oai_sets.pl script to > take into account the deletedbiblio table? My guess is that we can't (not straightforwardly at least). OAI sets are built from rules based on MARC informations. As marcxml is not kept in the deletedbiblio table, we can't use those rules to determine which deleted records belong to a given set. However, we could decide to keep the marcxml in the deletedbiblio table, or to add a field to the deletebiblio table with the oai set id the record used to belong to before its deletion.
Jonathan, i think the "no" option (and the whole syspref) is there to stick to the OAI standard: http://www.openarchives.org/OAI/openarchivesprotocol.html#DeletedRecords My understanding is that the transient, persistent or no status, is more a result of how you administer your database than one of the choice you made in this syspref. I'd say it's here so that the server communicates the right value to harvesters. Default value should be persistent, since the deletedbiblio table can only be emptied manually if the administrator does so. So transient would be "i do that from time to time" and 'no' would be "i never ever let anything go in this table".
Hm, I think initially including the deleted records when creating a set seems to go to far - you might probably not want those at this point. But once the set is created you want to inform about records removed from it. So changing the cronjob might not be necessary?
Created attachment 39993 [details] [review] Bug 3206 [Follow-up]: OAI repository deleted records support. - Fix QA.
I would like to get a signoff on the second patch.
Created attachment 40427 [details] [review] [SIGNED-OFF] Bug 3206 [Follow-up]: OAI repository deleted records support. - Fix QA. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> New pref in right order, new option 'no' on syspref, other fixes following comment #12 All seems to work No errors
Created attachment 40459 [details] [review] Bug 3206: OAI repository deleted records support. This patch allows Koha OAI repository to support deleted records. The OAI-PMH:DeletedRecord syspref is introduced and can be set to: - persistent (in case Koha's deletedbiblio table will never be emptied or truncated) - transient (in case Koha's deletedbiblio table might be emptied or truncated at some point) Test plan: - After applying the patch, test that: - Deleted records appear in ListRecords and ListIdentifiers requests. - Filter parameters (from, until, set and resumptionToken) still work and are applied to ListRecords and ListIdentifiers requests. - Identify request shows if the repository is considered persistent or transient, according to the OAI-PMH:DeletedRecord syspref. - Deleted records that used to belong to a set are still displayed in those sets and marked as deleted. - GetRecord requests work on deleted records, which are marked as deleted. Requests examples: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1 /cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=Identify Signed-off-by: Frederic Demians <f.demians@tamil.fr> It works in all situations described in the test plan. Great addition. Thanks. Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Created attachment 40460 [details] [review] Bug 3206 [Follow-up]: OAI repository deleted records support. - Fix QA. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> New pref in right order, new option 'no' on syspref, other fixes following comment #12 All seems to work No errors Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
(In reply to Katrin Fischer from comment #16) > Hm, I think initially including the deleted records when creating a set > seems to go to far - you might probably not want those at this point. But > once the set is created you want to inform about records removed from it. So > changing the cronjob might not be necessary? An answer is needed here.
I agree with Katrin, initially including deleted records seems a bit overkill to me.
Created attachment 40478 [details] [review] Bug 3206: OAI repository deleted records support. This patch allows Koha OAI repository to support deleted records. The OAI-PMH:DeletedRecord syspref is introduced and can be set to: - persistent (in case Koha's deletedbiblio table will never be emptied or truncated) - transient (in case Koha's deletedbiblio table might be emptied or truncated at some point) Test plan: - After applying the patch, test that: - Deleted records appear in ListRecords and ListIdentifiers requests. - Filter parameters (from, until, set and resumptionToken) still work and are applied to ListRecords and ListIdentifiers requests. - Identify request shows if the repository is considered persistent or transient, according to the OAI-PMH:DeletedRecord syspref. - Deleted records that used to belong to a set are still displayed in those sets and marked as deleted. - GetRecord requests work on deleted records, which are marked as deleted. Requests examples: /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&from=2015-02-20T11:08:33Z /cgi-bin/koha/oai.pl?verb=ListRecords&metadataPrefix=oai_dc&set=new_specSet1 /cgi-bin/koha/oai.pl?verb=GetRecord&identifier=KOHA-OAI-TEST:2&metadataPrefix=oai_dc /cgi-bin/koha/oai.pl?verb=Identify Signed-off-by: Frederic Demians <f.demians@tamil.fr> It works in all situations described in the test plan. Great addition. Thanks. Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org> Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Created attachment 40479 [details] [review] Bug 3206 [Follow-up]: OAI repository deleted records support. - Fix QA. Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> New pref in right order, new option 'no' on syspref, other fixes following comment #12 All seems to work No errors Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org> Signed-off-by: Jonathan Druart <jonathan.druart@koha-community.org>
Created attachment 40846 [details] [review] Bug 3206: (RM followup) DBIx updates Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Enahcement pushed to master. Good job Matthias!
Created attachment 40849 [details] [review] Bug 3206: (QA followup) missing comma on sysprefs.sql Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>