Koha needs to be able to harvest records from OAI-PMH repositories. This patch implements a cronjob-based OAI harvester, deliberately kept simple and consistent in its configuration with what we already have in Koha for other data sources, such as Z3950 and SRU. It is sponsored by KohaLa, its goals and implementation have been previously discussed on the koha and koha-devel mailing-lists, as well as in a presentation at KohaCon'23 in Helsinki: "Let's add an OAI-PMH harvester in Koha" by Sonia Bouis. Some of the code and ideas are based on a previous implementation of an OAI harvester by David Cook (See BZ 25905 and 10662).
Created attachment 160356 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa
Created attachment 160357 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update
This is great news! Once my January holidays are done, I can look more at these patches! At a glance, you will need to add unit tests for Koha::OAI::Client::Harvester before this can Pass QA (PERL17 in the coding guidelines), and that might require a bit of refactoring. (Missing unit tests are what ultimately bit me with bug 10662.) I think that we're not allowing function declarations in .pl scripts anymore either (for admin/oaiservers.pl) so that might need to be refactored. That said, I can't find a reference to it in the coding guidelines and the patches pass the QA scripts, so I could be wrong there. But other than that the patches are looking pretty good at a glance I think.
I'm having some trouble running /kohadevbox/koha/t/db_dependent/Koha/OAIServers.t as part of the QA scripts, but it might be my install. I'm going to try to refresh my ktd...
I updated my ktd and now I'm getting failures when running the QA tools, although I've got the unit test working. I notice there is a little typo in the pod in Koha/Import/Oaipmh/Authority.pm where it says "type" instead of "_type". Same with Koha/Import/Oaipmh/Biblio.pm and Koha/OaiServers.pm. Koha/OAI/Client/Harvester.pm is missing POD. I think the error for Koha/OaiServer.pm is false though. Not sure why that's getting a pod coverage error... I think the tidiness can also be ignored for the Schema files... -- qa -c 2 -v 2 testing 2 commit(s) (applied to ac02c1d 'ad Bug 35405: Remove noise from MARC:') Processing files before patches The PO directory has not been found. There is a problem in your Koha installation. at /kohadevbox/koha/Koha/I18N.pm line 197. |========================>| 27 / 27 (100.00%) Processing files after patches |========================>| 27 / 27 (100.00%) OK Koha/Import/Oaipmh/Authorities.pm FAIL Koha/Import/Oaipmh/Authority.pm FAIL pod coverage POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666 FAIL Koha/Import/Oaipmh/Biblio.pm FAIL pod coverage POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666 OK Koha/Import/Oaipmh/Biblios.pm FAIL Koha/OAI/Client/Harvester.pm FAIL pod coverage POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666 FAIL Koha/OaiServer.pm FAIL pod coverage POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666 OK Koha/OaiServers.pm FAIL Koha/Schema/Result/ImportOaipmhAuthority.pm FAIL pod coverage POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666 WARN tidiness The file is less tidy than before (bad/messy lines before: 0, now: 33) FAIL Koha/Schema/Result/ImportOaipmhBiblio.pm FAIL pod coverage POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666 WARN tidiness The file is less tidy than before (bad/messy lines before: 0, now: 28) FAIL Koha/Schema/Result/Oaiserver.pm FAIL pod coverage POD is missing for koha_object_class POD is missing for koha_objects_class WARN tidiness The file is less tidy than before (bad/messy lines before: 0, now: 24) OK admin/oaiservers.pl OK installer/data/mysql/atomicupdate/Bug-35659.pl OK installer/data/mysql/en/mandatory/sample_notices.yml OK installer/data/mysql/kohastructure.sql OK installer/data/mysql/mandatory/sysprefs.sql OK koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc OK koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc OK koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc OK koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt OK koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt OK koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl OK koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl OK koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl OK koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl OK koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl OK misc/cronjobs/harvest_oai.pl OK t/db_dependent/Koha/OAIServers.t Processing additional checks * Commit title does not start with 'Bug XXXXX: ' - 44b002f8cc Running tests (1) * Proving /kohadevbox/koha/t/db_dependent/Koha/OAIServers.t OK!
We are still missing supporters for this in the road map: https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*
(In reply to Katrin Fischer from comment #6) > We are still missing supporters for this in the road map: > https://wiki.koha-community.org/wiki/Road_map_24.05 *hint* I've just added myself. Hopefully someone else can jump on board too.
(In reply to Katrin Fischer from comment #6) > We are still missing supporters for this in the road map: > https://wiki.koha-community.org/wiki/Road_map_24.05 *hint* Add myself as supporter :)
Created attachment 160744 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa
Hi David! The typo in Koha/Import/Oaipmh/Authority.pm, Koha/Import/Oaipmh/Biblio.pm and Koha/OaiServers.pm has been fixed. POD has been added to Koha/OAI/Client/Harvester.pm > I think the error for Koha/OaiServer.pm is false though. Not sure why that's getting a pod coverage error... About those, ashimema told me on IRC: 15:07 ashimema if they're new classes you can safely ignore that.. 15:07 there's a bug in the POD detection for new clases > I think the tidiness can also be ignored for the Schema files... That's what I thought too, since they are auto-generated.
I'm in my last week of holiday mode, so I should be more available for this in the coming weeks. Thanks Matthias for looking at that. I'll take another look soon at the QA tools output another time. It looks like Koha::OAI::Client::Harvester is still missing unit tests though. I think that's going to be a QA blocker.
Comment on attachment 160744 [details] [review] Bug 35659: OAI Harvester Review of attachment 160744 [details] [review]: ----------------------------------------------------------------- ::: Koha/OAI/Client/Harvester.pm @@ +127,5 @@ > + printlog("ListMetadataFormats failed"); > + }; > + > + #my $response = $h->Identify; > + #print "Repository supports protocol version ", $response->version, "\n"; Looks like some debugging code that needs to be removed here
(In reply to David Cook from comment #11) > It looks like Koha::OAI::Client::Harvester is still missing unit tests > though. I think that's going to be a QA blocker. At a minimum, I think the code handling "$response->next" should be in separate unit tested functions. (In theory, the other earlier code within the "new" constructor could probably be moved into an "init" function as well. But that is harder to test since it interacts with a real website. We could create a fake website but I think that would be a bit above and beyond.)
Created attachment 161813 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa
(In reply to David Cook from comment #13) > (In reply to David Cook from comment #11) > > It looks like Koha::OAI::Client::Harvester is still missing unit tests > > though. I think that's going to be a QA blocker. > > At a minimum, I think the code handling "$response->next" should be in > separate unit tested functions. > Hi! I've added unit tests for record processing.
(In reply to Matthias Meusburger from comment #15) > Hi! I've added unit tests for record processing. Thanks, Matthias! I've taken a quick glance at the unit tests and they're looking good! I've got a bit on my plate at the moment, but I'll come back to look at this soon! I think it would be good to find someone else to Sign Off on it, so that I can QA it.
Hi, I get an error when I try to apply the patch: Bug 35659 - OAI Harvester 160357 - DO NOT PUSH: Bug 35659 DBIC Update 161813 - Bug 35659: OAI Harvester Apply? [(y)es, (n)o, (i)nteractive] y Applying: DO NOT PUSH: Bug 35659 DBIC Update Applying: Bug 35659: OAI Harvester Using index info to reconstruct a base tree... M installer/data/mysql/mandatory/sysprefs.sql M installer/data/mysql/mandatory/userpermissions.sql M koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc M koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt Falling back to patching base and 3-way merge... Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc Auto-merging installer/data/mysql/mandatory/userpermissions.sql Auto-merging installer/data/mysql/mandatory/sysprefs.sql CONFLICT (content): Merge conflict in installer/data/mysql/mandatory/sysprefs.sql error: Failed to merge in the changes.
Created attachment 161960 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa
(In reply to Michal Denar from comment #17) > Hi, > I get an error when I try to apply the patch: > Hi Michal, sorry about that. The patch has been rebased to master, it should be ok now.
Hi, this patch works for me. I don't know why, but I can't get it to unsign the signed patch and switch to SO. I've tried several times, but without success. Harvester works and it is possible to switch to signed off.
I get error: "Failed to update bug 35659, status=200" BZ login credentials are correct.
Created attachment 162004 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: John Doe <you@example.com>
Created attachment 162005 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: John Doe <you@example.com>
Created attachment 162006 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: John Doe <you@example.com>
Created attachment 162007 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: John Doe <you@example.com>
Created attachment 162008 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: Michal <black232gmail.com>
Created attachment 162009 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: John Doe <you@example.com> Signed-off-by: Michal <black232gmail.com>
Created attachment 162010 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: Michal <black23@gmail.com>
Created attachment 162011 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: John Doe <you@example.com> Signed-off-by: Michal <black232gmail.com> Signed-off-by: John Doe <you@example.com>
Created attachment 162527 [details] Screenshot - Accented characters in the OPAC
Just wanted to point out that I tried to harvest bibliographic records from the first repository suggested in the Test plan (http://staroai.theses.fr/OAIHandler, oai_dc data format) in a MARC21 Koha instance (therefore using OAIDCtoMARC21XML.xsl to transform the records prior to importing), and when I ran the OAI harvester script in verbose mode I got many lines similar to the following in the output: no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. The script did actually finish and managed to add 91 records, but I noticed that in the biblio details page in the OPAC some (not all) words containing accented characters did not appear correctly. I have uploaded a screenshot that illustrates this. Upon further investigation, it looks like the script *does* manage to populate database fields such as biblio.title and biblio.abstract with the correctly accented characters, but some MARC fields (such as 245 - Title, and 6xx - Subjects) in the MARCXML representation in biblio_metadata.metadata still contain weird characters wherever accents exist. The fix is to apply Franck Theeten's solution from Bug 16488 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24) and modify the value of the MARC leader's 10th character to 'a' in the XSLT. Then, the accented characters get imported properly and the records appear correctly in the OPAC.
Created attachment 162528 [details] [review] Bug 35659: (follow-up) Better handling of accented characters If you try to harvest bibliographic records from a UNIMARC OAI repository (using oai_dc data format) in a MARC21 Koha instance and run the OAI harvester script in verbose mode, you may get lines similar to the following in the output: no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. When looking at the imported records' biblio details page in the OPAC, most words containing accented characters will not appear correctly. The fix is to apply Franck Theeten's solution from Bug 16488 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24) and modify the value of the MARC leader's 10th character to 'a' in the XSLT that transforms the UNIMARC OAI records into MARC21 XML. Then, the accented characters get imported properly and the records appear correctly in the OPAC. Test plan: 0) Without this patch, running the OAI harvesting script in verbose mode produces many warnings, and garbled characters appear in the OPAC biblio details page wherever accented characters are in use. 1) Apply this patch. 2) Re-run the OAI harvesting script in verbose + force mode (force mode is required to ignore record datestamps from previous runs): misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f This time there should be no warnings printed on your screen, and any characters with accents in the updated records should look OK in the OPAC. Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>
Thank you Andreas for your feedback! Not sure if I can sign-off your follow-up myself, since I'm the author of the main patch, but I can confirm that it works as intended: warnings are gone, and encoding is now correct.
Created attachment 162692 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: Michal Denar <black23@gmail.com>
Created attachment 162693 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: John Doe <you@example.com> Signed-off-by: Michal <black232gmail.com> Signed-off-by: John Doe <you@example.com> Signed-off-by: Michal Denar <black23@gmail.com>
Created attachment 162694 [details] [review] Bug 35659: (follow-up) Better handling of accented characters If you try to harvest bibliographic records from a UNIMARC OAI repository (using oai_dc data format) in a MARC21 Koha instance and run the OAI harvester script in verbose mode, you may get lines similar to the following in the output: no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. When looking at the imported records' biblio details page in the OPAC, most words containing accented characters will not appear correctly. The fix is to apply Franck Theeten's solution from Bug 16488 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24) and modify the value of the MARC leader's 10th character to 'a' in the XSLT that transforms the UNIMARC OAI records into MARC21 XML. Then, the accented characters get imported properly and the records appear correctly in the OPAC. Test plan: 0) Without this patch, running the OAI harvesting script in verbose mode produces many warnings, and garbled characters appear in the OPAC biblio details page wherever accented characters are in use. 1) Apply this patch. 2) Re-run the OAI harvesting script in verbose + force mode (force mode is required to ignore record datestamps from previous runs): misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f This time there should be no warnings printed on your screen, and any characters with accents in the updated records should look OK in the OPAC. Thanks-to: Franck Theeten <franck.theeten@africamuseum.be> Signed-off-by: Michal Denar <black23@gmail.com>
(obsoleted duplicate old version of a patch)
Adding a server doesn't work due to not complying with CSRF (Bug 34478): > Programming error - op 'add_validated' must start with 'cud-' for POST http://localhost:8081/intranet/admin/oaiservers.pl (referer: http://localhost:8081/cgi-bin/koha/admin/oaiservers.pl?op=add) From Koha/Middleware/CSRF.pm
Created attachment 163527 [details] [review] Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF
Thanks for your feedback Tuxayo!
Comment on attachment 162693 [details] [review] Bug 35659: OAI Harvester Review of attachment 162693 [details] [review]: ----------------------------------------------------------------- Overall this is working really well. OAI repositories mentioned in the test plan can take some time to respond (or fail with a timeout) but this is handled well. However, there are some minor things to fix (see the rest of the review). ::: Koha/OAI/Client/Harvester.pm @@ +79,5 @@ > +C<$force> force harvesting (ignore records datestamps) > + > +=cut > + > +sub new { The `new` method doesn't instantiate an object. Instead it sets global variables and returns the class name. I think it's confusing, as all other `new` methods in Koha instantiate an object. @@ +142,5 @@ > + > + if ($days) { > + > + # Change this to yyyy-mm-dd > + my $dt_today = dt_from_string(); This variable is not used anywhere. @@ +312,5 @@ > + $imported_record->update( > + { > + datestamp => $imported_record->datestamp, > + } > + ); This looks like it does nothing (set "record's datestamp" to "record's datestamp"). Shouldn't it be $oai_record->datestamp (or now if incoming oai record does not have a datestamp) ?
Side note: Koha::OAI::Client::Harvester uses cronlogaction, which makes it tied to the script using it (misc/cronjobs/harvest_oai.pl). As this script is the only way to use the harvester, I think it's okay to keep it like that for now, but it will have to be changed if we want to use the harvester elsewhere (a background job for instance)
It looks like CSRF validation does not work: if I manually change the token in the deletion form and then try to delete the corresponding OAI repository, the repository is deleted.
(In reply to Julian Maurice from comment #43) > It looks like CSRF validation does not work: if I manually change the token > in the deletion form and then try to delete the corresponding OAI > repository, the repository is deleted. Please ignore this comment, it was caused by a bad configuration on my instance
Created attachment 164586 [details] [review] Bug 35659: (QA follow-up)
Created attachment 164590 [details] [review] Bug 35659: (QA follow-up)
Created attachment 164602 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 164603 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 164604 [details] [review] Bug 35659: (follow-up) Better handling of accented characters If you try to harvest bibliographic records from a UNIMARC OAI repository (using oai_dc data format) in a MARC21 Koha instance and run the OAI harvester script in verbose mode, you may get lines similar to the following in the output: no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. When looking at the imported records' biblio details page in the OPAC, most words containing accented characters will not appear correctly. The fix is to apply Franck Theeten's solution from Bug 16488 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24) and modify the value of the MARC leader's 10th character to 'a' in the XSLT that transforms the UNIMARC OAI records into MARC21 XML. Then, the accented characters get imported properly and the records appear correctly in the OPAC. Test plan: 0) Without this patch, running the OAI harvesting script in verbose mode produces many warnings, and garbled characters appear in the OPAC biblio details page wherever accented characters are in use. 1) Apply this patch. 2) Re-run the OAI harvesting script in verbose + force mode (force mode is required to ignore record datestamps from previous runs): misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f This time there should be no warnings printed on your screen, and any characters with accents in the updated records should look OK in the OPAC. Thanks-to: Franck Theeten <franck.theeten@africamuseum.be> Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 164605 [details] [review] Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 164606 [details] [review] Bug 35659: (QA follow-up) Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Created attachment 164618 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 164619 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 164620 [details] [review] Bug 35659: (follow-up) Better handling of accented characters If you try to harvest bibliographic records from a UNIMARC OAI repository (using oai_dc data format) in a MARC21 Koha instance and run the OAI harvester script in verbose mode, you may get lines similar to the following in the output: no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. When looking at the imported records' biblio details page in the OPAC, most words containing accented characters will not appear correctly. The fix is to apply Franck Theeten's solution from Bug 16488 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24) and modify the value of the MARC leader's 10th character to 'a' in the XSLT that transforms the UNIMARC OAI records into MARC21 XML. Then, the accented characters get imported properly and the records appear correctly in the OPAC. Test plan: 0) Without this patch, running the OAI harvesting script in verbose mode produces many warnings, and garbled characters appear in the OPAC biblio details page wherever accented characters are in use. 1) Apply this patch. 2) Re-run the OAI harvesting script in verbose + force mode (force mode is required to ignore record datestamps from previous runs): misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f This time there should be no warnings printed on your screen, and any characters with accents in the updated records should look OK in the OPAC. Thanks-to: Franck Theeten <franck.theeten@africamuseum.be> Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 164621 [details] [review] Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 164622 [details] [review] Bug 35659: (QA follow-up) Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
+ $dbh->do(q{DROP TABLE IF EXISTS `oaiservers`;}); You shouldn't drop the table but create it only if it does not exist.
Created attachment 165199 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
(In reply to Jonathan Druart from comment #57) > + $dbh->do(q{DROP TABLE IF EXISTS `oaiservers`;}); > > You shouldn't drop the table but create it only if it does not exist. Thanks Joubu, this is fixed.
Some comments: * Why is the `oaiservers` table name used instead of `oai_servers`? (same for oaiservers.{ps|tt}) * Why we sometimes call it `oai` and sometimes `oaipmh`? * OaiServer should either be OAI::Server or OAIServer. * Column names look old-school IMHO (`id`vs `oai_server_id`) * Can we pick names that match biblio_metadata columns for example? Like `format` and `schema` instead of `recordtype` * Is `datestamp` some OAI-PMH specific naming convention? * `repository` varchar(255) NOT NULL COMMENT 'OAI repository'? Shouldn't this be a FK on the `oaiservers` table? This feels important as we are also adding 'record sources' on another bug, and we will eventually want to set rules for OAI-PMH imported records. * Can the OAI harvester be thought as a background job in the mid/long term? In that case, it feels weird that Koha::OAI::Client::Harvester just prints things. It might be better to -for example- accumulate them using Koha::Object->add_message and have the caller decide what to do? (so harvest_oai.pl decides what to do) or it could just be a callback function for dealing with each step's output.
> * Column names look old-school IMHO (`id`vs `oai_server_id`) That's a definitely an issue: https://wiki.koha-community.org/wiki/Coding_Guidelines#SQL7:_Primary_keys «primary key name must be tablename_id (color_id for example) » Also it seems for new columns we are now using snake_case: https://wiki.koha-community.org/wiki/Database_updates#How_to_write_an_atomicupdate_file «Example 2: Adding a new column» Except for foreign keys like biblionumber > * Why is the `oaiservers` table name used instead of `oai_servers`? (same for oaiservers.{ps|tt}) +1 for consistency with how new stuff is usually named. --- Code: > sub ServerSearch This rule should also apply on new all new files outside C4: https://wiki.koha-community.org/wiki/Coding_Guidelines#PERL9:_Subroutine_naming_conventions «subroutines in the Koha namespace should be snake case» --- String changes: #: installer/data/mysql/en/mandatory/sample_notices.yml:letter:1:row:98:mul:4 msgid "Set: %s" #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt:92 msgid "Set: " +#: koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt:136 +#, fuzzy, c-format +msgid "Set" They need context for translators: https://wiki.koha-community.org/wiki/Internationalization,_plural_forms,_context,_and_more_RFC#More_on_context And using tp() is an opportunity to deduplicate "Set: " and "Set" +#: koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt:126 +#, c-format +msgid "%s You searched for record %s %s You searched for %s %s " This string should be split. Either via t() or <span> --- I reattached the patches to fix the order that broke applying.
Created attachment 165594 [details] [review] DO NOT PUSH: Bug 35659 DBIC Update Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 165595 [details] [review] Bug 35659: OAI Harvester Test plan: - Run updatedatabase.pl - Apply the DBIC update patch - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl) If the repository dataformat is in oai_dc, you can use the sample XSLT files: For Unimarc: OAIDCtoUNIMARCXML.xsl for bibliographic records AuthOAIDCtoUNIMARCXML.xsl for authorities For Marc21: OAIDCtoMARC21XML.xsl for bibliographic records AuthOAIDCtoMARC21XML.xsl for authorities If the repository dataformat is in marc-xml, you can use OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21. - Some repositories to test with: Bibliographic records, oai_dc: Endpoint: http://staroai.theses.fr/OAIHandler Set: ddc:260 XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl Authorities, oai_dc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl or XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl Authorities, marc-xml, Unimarc: Endpoint: https://www.idref.fr/OAI/oai.jsp Set: a XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx Version 4.xx is needed. See Bug 17704 for updating Koha default version to 4.xx - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips. - start a harvest with the following command: misc/cronjobs/harvest_oai.pl This script starts an OAI Harvest This script has the following parameters : -h --help: this message -v --verbose (enable output to stdout) -r --repository: id of the OAI repository -d --days: number of days to harvest from (optional) -l --list: list the OAI repositories -f --force: force harvesting (ignore records datestamps) (optional) Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f - Check the logs in the Log Viewer (tools/viewlog.pl) (CronjobLog syspref must be activated) Sponsored-by: KohaLa Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 165596 [details] [review] Bug 35659: (follow-up) Better handling of accented characters If you try to harvest bibliographic records from a UNIMARC OAI repository (using oai_dc data format) in a MARC21 Koha instance and run the OAI harvester script in verbose mode, you may get lines similar to the following in the output: no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308. When looking at the imported records' biblio details page in the OPAC, most words containing accented characters will not appear correctly. The fix is to apply Franck Theeten's solution from Bug 16488 (https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24) and modify the value of the MARC leader's 10th character to 'a' in the XSLT that transforms the UNIMARC OAI records into MARC21 XML. Then, the accented characters get imported properly and the records appear correctly in the OPAC. Test plan: 0) Without this patch, running the OAI harvesting script in verbose mode produces many warnings, and garbled characters appear in the OPAC biblio details page wherever accented characters are in use. 1) Apply this patch. 2) Re-run the OAI harvesting script in verbose + force mode (force mode is required to ignore record datestamps from previous runs): misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f This time there should be no warnings printed on your screen, and any characters with accents in the updated records should look OK in the OPAC. Thanks-to: Franck Theeten <franck.theeten@africamuseum.be> Signed-off-by: Michal Denar <black23@gmail.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 165597 [details] [review] Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 165598 [details] [review] Bug 35659: (QA follow-up) Signed-off-by: Julian Maurice <julian.maurice@biblibre.com> Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>