Bug 35659 - OAI Harvester
Summary: OAI Harvester
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Matthias Meusburger
QA Contact: Testopia
URL:
Keywords: no-sandbox, roadmap_24_05
Depends on:
Blocks:
 
Reported: 2023-12-28 15:43 UTC by Matthias Meusburger
Modified: 2024-04-26 04:16 UTC (History)
13 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 35659: OAI Harvester (99.13 KB, patch)
2023-12-28 16:00 UTC, Matthias Meusburger
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.11 KB, patch)
2023-12-28 16:00 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (99.56 KB, patch)
2024-01-10 09:35 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (102.98 KB, patch)
2024-02-07 15:11 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (102.95 KB, patch)
2024-02-09 09:20 UTC, Matthias Meusburger
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.15 KB, patch)
2024-02-10 20:09 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.00 KB, patch)
2024-02-10 20:09 UTC, Michal Denar
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.15 KB, patch)
2024-02-10 20:10 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.00 KB, patch)
2024-02-10 20:10 UTC, Michal Denar
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.19 KB, patch)
2024-02-10 20:12 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.04 KB, patch)
2024-02-10 20:12 UTC, Michal Denar
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.23 KB, patch)
2024-02-10 20:33 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.08 KB, patch)
2024-02-10 20:33 UTC, Michal Denar
Details | Diff | Splinter Review
Screenshot - Accented characters in the OPAC (90.94 KB, image/png)
2024-02-28 07:32 UTC, Andreas Roussos
Details
Bug 35659: (follow-up) Better handling of accented characters (2.79 KB, patch)
2024-02-28 07:35 UTC, Andreas Roussos
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.28 KB, patch)
2024-03-01 20:24 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.12 KB, patch)
2024-03-01 20:24 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: (follow-up) Better handling of accented characters (2.84 KB, patch)
2024-03-01 20:24 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF (6.98 KB, patch)
2024-03-20 13:37 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: (QA follow-up) (6.21 KB, patch)
2024-04-10 08:36 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: (QA follow-up) (14.81 KB, patch)
2024-04-10 09:12 UTC, Matthias Meusburger
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.22 KB, patch)
2024-04-10 11:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.06 KB, patch)
2024-04-10 11:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 35659: (follow-up) Better handling of accented characters (2.90 KB, patch)
2024-04-10 11:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF (7.04 KB, patch)
2024-04-10 11:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 35659: (QA follow-up) (14.87 KB, patch)
2024-04-10 11:33 UTC, Julian Maurice
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.28 KB, patch)
2024-04-10 12:31 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.17 KB, patch)
2024-04-10 12:31 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35659: (follow-up) Better handling of accented characters (2.96 KB, patch)
2024-04-10 12:32 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF (7.10 KB, patch)
2024-04-10 12:32 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35659: (QA follow-up) (14.93 KB, patch)
2024-04-10 12:32 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (102.96 KB, patch)
2024-04-19 13:42 UTC, Matthias Meusburger
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.27 KB, patch)
2024-04-26 04:15 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (102.96 KB, patch)
2024-04-26 04:15 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 35659: (follow-up) Better handling of accented characters (2.95 KB, patch)
2024-04-26 04:16 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF (7.09 KB, patch)
2024-04-26 04:16 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 35659: (QA follow-up) (14.92 KB, patch)
2024-04-26 04:16 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Meusburger 2023-12-28 15:43:23 UTC
Koha needs to be able to harvest records from OAI-PMH repositories.

This patch implements a cronjob-based OAI harvester, deliberately kept simple and consistent in its configuration with what we already have in Koha for other data sources, such as Z3950 and SRU.

It is sponsored by KohaLa, its goals and implementation have been previously discussed on the koha and koha-devel mailing-lists, as well as in a presentation at KohaCon'23 in Helsinki: "Let's add an OAI-PMH harvester in Koha" by Sonia Bouis.

Some of the code and ideas are based on a previous implementation of an OAI harvester by David Cook (See BZ 25905 and 10662).
Comment 1 Matthias Meusburger 2023-12-28 16:00:25 UTC
Created attachment 160356 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 2 Matthias Meusburger 2023-12-28 16:00:44 UTC
Created attachment 160357 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update
Comment 3 David Cook 2024-01-02 01:53:32 UTC
This is great news! Once my January holidays are done, I can look more at these patches!

At a glance, you will need to add unit tests for Koha::OAI::Client::Harvester before this can Pass QA (PERL17 in the coding guidelines), and that might require a bit of refactoring. (Missing unit tests are what ultimately bit me with bug 10662.)

I think that we're not allowing function declarations in .pl scripts anymore either (for admin/oaiservers.pl) so that might need to be refactored. That said, I can't find a reference to it in the coding guidelines and the patches pass the QA scripts, so I could be wrong there. 

But other than that the patches are looking pretty good at a glance I think.
Comment 4 David Cook 2024-01-02 01:54:12 UTC
I'm having some trouble running /kohadevbox/koha/t/db_dependent/Koha/OAIServers.t as part of the QA scripts, but it might be my install. I'm going to try to refresh my ktd...
Comment 5 David Cook 2024-01-02 02:10:49 UTC
I updated my ktd and now I'm getting failures when running the QA tools, although I've got the unit test working.

I notice there is a little typo in the pod in Koha/Import/Oaipmh/Authority.pm where it says "type" instead of "_type". Same with Koha/Import/Oaipmh/Biblio.pm and Koha/OaiServers.pm. 

Koha/OAI/Client/Harvester.pm is missing POD. 

I think the error for Koha/OaiServer.pm is false though. Not sure why that's getting a pod coverage error... 

I think the tidiness can also be ignored for the Schema files...

--

qa -c 2 -v 2
testing 2 commit(s) (applied to ac02c1d 'ad Bug 35405: Remove noise from MARC:')

Processing files before patches
The PO directory has not been found. There is a problem in your Koha installation. at /kohadevbox/koha/Koha/I18N.pm line 197.
|========================>| 27 / 27 (100.00%)
Processing files after patches
|========================>| 27 / 27 (100.00%)

 OK     Koha/Import/Oaipmh/Authorities.pm

 FAIL   Koha/Import/Oaipmh/Authority.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 FAIL   Koha/Import/Oaipmh/Biblio.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 OK     Koha/Import/Oaipmh/Biblios.pm

 FAIL   Koha/OAI/Client/Harvester.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 FAIL   Koha/OaiServer.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 OK     Koha/OaiServers.pm

 FAIL   Koha/Schema/Result/ImportOaipmhAuthority.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 33)

 FAIL   Koha/Schema/Result/ImportOaipmhBiblio.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 28)

 FAIL   Koha/Schema/Result/Oaiserver.pm
   FAIL   pod coverage
                POD is missing for koha_object_class
                POD is missing for koha_objects_class
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 24)

 OK     admin/oaiservers.pl

 OK     installer/data/mysql/atomicupdate/Bug-35659.pl

 OK     installer/data/mysql/en/mandatory/sample_notices.yml

 OK     installer/data/mysql/kohastructure.sql

 OK     installer/data/mysql/mandatory/sysprefs.sql

 OK     koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc

 OK     koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc

 OK     koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc

 OK     koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt

 OK     koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 OK     misc/cronjobs/harvest_oai.pl

 OK     t/db_dependent/Koha/OAIServers.t


Processing additional checks

        * Commit title does not start with 'Bug XXXXX: ' - 44b002f8cc

Running tests (1)
        * Proving /kohadevbox/koha/t/db_dependent/Koha/OAIServers.t OK!
Comment 6 Katrin Fischer 2024-01-02 22:18:36 UTC
We are still missing supporters for this in the road map:
https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*
Comment 7 David Cook 2024-01-02 22:25:40 UTC
(In reply to Katrin Fischer from comment #6)
> We are still missing supporters for this in the road map:
> https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*

I've just added myself. Hopefully someone else can jump on board too.
Comment 8 Koha Team University Lyon 3 2024-01-06 11:54:46 UTC
(In reply to Katrin Fischer from comment #6)
> We are still missing supporters for this in the road map:
> https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*

Add myself as supporter :)
Comment 9 Matthias Meusburger 2024-01-10 09:35:36 UTC
Created attachment 160744 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 10 Matthias Meusburger 2024-01-10 09:45:34 UTC
Hi David!

The typo in Koha/Import/Oaipmh/Authority.pm, Koha/Import/Oaipmh/Biblio.pm and Koha/OaiServers.pm has been fixed.

POD has been added to Koha/OAI/Client/Harvester.pm

> I think the error for Koha/OaiServer.pm is false though. Not sure why that's getting a pod coverage error... 

About those, ashimema told me on IRC: 

15:07	ashimema	if they're new classes you can safely ignore that..
15:07		there's a bug in the POD detection for new clases

> I think the tidiness can also be ignored for the Schema files...

That's what I thought too, since they are auto-generated.
Comment 11 David Cook 2024-01-22 22:44:53 UTC
I'm in my last week of holiday mode, so I should be more available for this in the coming weeks.

Thanks Matthias for looking at that. I'll take another look soon at the QA tools output another time.

It looks like Koha::OAI::Client::Harvester is still missing unit tests though. I think that's going to be a QA blocker.
Comment 12 David Cook 2024-01-22 22:45:42 UTC
Comment on attachment 160744 [details] [review]
Bug 35659: OAI Harvester

Review of attachment 160744 [details] [review]:
-----------------------------------------------------------------

::: Koha/OAI/Client/Harvester.pm
@@ +127,5 @@
> +        printlog("ListMetadataFormats failed");
> +    };
> +
> +    #my $response = $h->Identify;
> +    #print "Repository supports protocol version ", $response->version, "\n";

Looks like some debugging code that needs to be removed here
Comment 13 David Cook 2024-01-22 22:58:37 UTC
(In reply to David Cook from comment #11)
> It looks like Koha::OAI::Client::Harvester is still missing unit tests
> though. I think that's going to be a QA blocker.

At a minimum, I think the code handling "$response->next" should be in separate unit tested functions. 

(In theory, the other earlier code within the "new" constructor could probably be moved into an "init" function as well. But that is harder to test since it interacts with a real website. We could create a fake website but I think that would be a bit above and beyond.)
Comment 14 Matthias Meusburger 2024-02-07 15:11:05 UTC
Created attachment 161813 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 15 Matthias Meusburger 2024-02-07 15:12:21 UTC
(In reply to David Cook from comment #13)
> (In reply to David Cook from comment #11)
> > It looks like Koha::OAI::Client::Harvester is still missing unit tests
> > though. I think that's going to be a QA blocker.
> 
> At a minimum, I think the code handling "$response->next" should be in
> separate unit tested functions. 
> 

Hi! I've added unit tests for record processing.
Comment 16 David Cook 2024-02-07 22:38:10 UTC
(In reply to Matthias Meusburger from comment #15)
> Hi! I've added unit tests for record processing.

Thanks, Matthias! I've taken a quick glance at the unit tests and they're looking good! 

I've got a bit on my plate at the moment, but I'll come back to look at this soon! 

I think it would be good to find someone else to Sign Off on it, so that I can QA it.
Comment 17 Michal Denar 2024-02-08 20:38:27 UTC
Hi,
I get an error when I try to apply the patch:

Bug 35659 - OAI Harvester

160357 - DO NOT PUSH: Bug 35659 DBIC Update
161813 - Bug 35659: OAI Harvester

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: DO NOT PUSH: Bug 35659 DBIC Update
Applying: Bug 35659: OAI Harvester
Using index info to reconstruct a base tree...
M	installer/data/mysql/mandatory/sysprefs.sql
M	installer/data/mysql/mandatory/userpermissions.sql
M	koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc
Auto-merging installer/data/mysql/mandatory/userpermissions.sql
Auto-merging installer/data/mysql/mandatory/sysprefs.sql
CONFLICT (content): Merge conflict in installer/data/mysql/mandatory/sysprefs.sql
error: Failed to merge in the changes.
Comment 18 Matthias Meusburger 2024-02-09 09:20:44 UTC
Created attachment 161960 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 19 Matthias Meusburger 2024-02-09 09:22:17 UTC
(In reply to Michal Denar from comment #17)
> Hi,
> I get an error when I try to apply the patch:
> 

Hi Michal, sorry about that. The patch has been rebased to master, it should be ok now.
Comment 20 Michal Denar 2024-02-09 21:31:42 UTC
Hi,
this patch works for me. I don't know why, but I can't get it to unsign the signed patch and switch to SO. I've tried several times, but without success. Harvester works and it is possible to switch to signed off.
Comment 21 Michal Denar 2024-02-09 21:34:45 UTC
I get error: "Failed to update bug 35659, status=200"
BZ login credentials are correct.
Comment 22 Michal Denar 2024-02-10 20:09:00 UTC
Created attachment 162004 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: John Doe <you@example.com>
Comment 23 Michal Denar 2024-02-10 20:09:03 UTC
Created attachment 162005 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Comment 24 Michal Denar 2024-02-10 20:10:00 UTC
Created attachment 162006 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: John Doe <you@example.com>
Comment 25 Michal Denar 2024-02-10 20:10:04 UTC
Created attachment 162007 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Comment 26 Michal Denar 2024-02-10 20:12:34 UTC
Created attachment 162008 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal <black232gmail.com>
Comment 27 Michal Denar 2024-02-10 20:12:40 UTC
Created attachment 162009 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Signed-off-by: Michal <black232gmail.com>
Comment 28 Michal Denar 2024-02-10 20:33:32 UTC
Created attachment 162010 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal <black23@gmail.com>
Comment 29 Michal Denar 2024-02-10 20:33:35 UTC
Created attachment 162011 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Signed-off-by: Michal <black232gmail.com>
Signed-off-by: John Doe <you@example.com>
Comment 30 Andreas Roussos 2024-02-28 07:32:19 UTC
Created attachment 162527 [details]
Screenshot - Accented characters in the OPAC
Comment 31 Andreas Roussos 2024-02-28 07:34:07 UTC
Just wanted to point out that I tried to harvest bibliographic
records from the first repository suggested in the Test plan
(http://staroai.theses.fr/OAIHandler, oai_dc data format) in a
MARC21 Koha instance (therefore using OAIDCtoMARC21XML.xsl to
transform the records prior to importing), and when I ran the
OAI harvester script in verbose mode I got many lines similar
to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

The script did actually finish and managed to add 91 records,
but I noticed that in the biblio details page in the OPAC some
(not all) words containing accented characters did not appear
correctly. I have uploaded a screenshot that illustrates this.

Upon further investigation, it looks like the script *does*
manage to populate database fields such as biblio.title and
biblio.abstract with the correctly accented characters, but
some MARC fields (such as 245 - Title, and 6xx - Subjects)
in the MARCXML representation in biblio_metadata.metadata
still contain weird characters wherever accents exist.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.
Comment 32 Andreas Roussos 2024-02-28 07:35:14 UTC
Created attachment 162528 [details] [review]
Bug 35659: (follow-up) Better handling of accented characters

If you try to harvest bibliographic records from a UNIMARC OAI
repository (using oai_dc data format) in a MARC21 Koha instance
and run the OAI harvester script in verbose mode, you may get
lines similar to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

When looking at the imported records' biblio details page in
the OPAC, most words containing accented characters will not
appear correctly.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT that transforms the UNIMARC OAI records
into MARC21 XML. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.

Test plan:

0) Without this patch, running the OAI harvesting script in
   verbose mode produces many warnings, and garbled characters
   appear in the OPAC biblio details page wherever accented
   characters are in use.

1) Apply this patch.

2) Re-run the OAI harvesting script in verbose + force mode
   (force mode is required to ignore record datestamps from
   previous runs):

   misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f

   This time there should be no warnings printed on your
   screen, and any characters with accents in the updated
   records should look OK in the OPAC.

Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>
Comment 33 Matthias Meusburger 2024-03-01 10:27:42 UTC
Thank you Andreas for your feedback!

Not sure if I can sign-off your follow-up myself, since I'm the author of the main patch, but I can confirm that it works as intended: warnings are gone, and encoding is now correct.
Comment 34 Michal Denar 2024-03-01 20:24:36 UTC
Created attachment 162692 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 35 Michal Denar 2024-03-01 20:24:40 UTC
Created attachment 162693 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Signed-off-by: Michal <black232gmail.com>
Signed-off-by: John Doe <you@example.com>
Signed-off-by: Michal Denar <black23@gmail.com>
Comment 36 Michal Denar 2024-03-01 20:24:43 UTC
Created attachment 162694 [details] [review]
Bug 35659: (follow-up) Better handling of accented characters

If you try to harvest bibliographic records from a UNIMARC OAI
repository (using oai_dc data format) in a MARC21 Koha instance
and run the OAI harvester script in verbose mode, you may get
lines similar to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

When looking at the imported records' biblio details page in
the OPAC, most words containing accented characters will not
appear correctly.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT that transforms the UNIMARC OAI records
into MARC21 XML. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.

Test plan:

0) Without this patch, running the OAI harvesting script in
   verbose mode produces many warnings, and garbled characters
   appear in the OPAC biblio details page wherever accented
   characters are in use.

1) Apply this patch.

2) Re-run the OAI harvesting script in verbose + force mode
   (force mode is required to ignore record datestamps from
   previous runs):

   misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f

   This time there should be no warnings printed on your
   screen, and any characters with accents in the updated
   records should look OK in the OPAC.

Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>
Signed-off-by: Michal Denar <black23@gmail.com>
Comment 37 Victor Grousset/tuxayo 2024-03-16 01:02:17 UTC
(obsoleted duplicate old version of a patch)
Comment 38 Victor Grousset/tuxayo 2024-03-17 03:21:19 UTC
Adding a server doesn't work due to not complying with CSRF (Bug 34478):

> Programming error - op 'add_validated' must start with 'cud-' for POST http://localhost:8081/intranet/admin/oaiservers.pl (referer: http://localhost:8081/cgi-bin/koha/admin/oaiservers.pl?op=add)

From Koha/Middleware/CSRF.pm
Comment 39 Matthias Meusburger 2024-03-20 13:37:10 UTC
Created attachment 163527 [details] [review]
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF
Comment 40 Matthias Meusburger 2024-03-20 13:38:04 UTC
Thanks for your feedback Tuxayo!
Comment 41 Julian Maurice 2024-04-08 14:12:12 UTC
Comment on attachment 162693 [details] [review]
Bug 35659: OAI Harvester

Review of attachment 162693 [details] [review]:
-----------------------------------------------------------------

Overall this is working really well. OAI repositories mentioned in the test plan can take some time to respond (or fail with a timeout) but this is handled well.

However, there are some minor things to fix (see the rest of the review).

::: Koha/OAI/Client/Harvester.pm
@@ +79,5 @@
> +C<$force> force harvesting (ignore records datestamps)
> +
> +=cut
> +
> +sub new {

The `new` method doesn't instantiate an object. Instead it sets global variables and returns the class name.
I think it's confusing, as all other `new` methods in Koha instantiate an object.

@@ +142,5 @@
> +
> +    if ($days) {
> +
> +        # Change this to yyyy-mm-dd
> +        my $dt_today = dt_from_string();

This variable is not used anywhere.

@@ +312,5 @@
> +                $imported_record->update(
> +                    {
> +                        datestamp => $imported_record->datestamp,
> +                    }
> +                );

This looks like it does nothing (set "record's datestamp" to "record's datestamp").
Shouldn't it be $oai_record->datestamp (or now if incoming oai record does not have a datestamp) ?
Comment 42 Julian Maurice 2024-04-08 14:16:12 UTC
Side note: Koha::OAI::Client::Harvester uses cronlogaction, which makes it tied to the script using it (misc/cronjobs/harvest_oai.pl).
As this script is the only way to use the harvester, I think it's okay to keep it like that for now, but it will have to be changed if we want to use the harvester elsewhere (a background job for instance)
Comment 43 Julian Maurice 2024-04-08 14:53:00 UTC Comment hidden (obsolete)
Comment 44 Julian Maurice 2024-04-10 08:28:50 UTC
(In reply to Julian Maurice from comment #43)
> It looks like CSRF validation does not work: if I manually change the token
> in the deletion form and then try to delete the corresponding OAI
> repository, the repository is deleted.

Please ignore this comment, it was caused by a bad configuration on my instance
Comment 45 Matthias Meusburger 2024-04-10 08:36:36 UTC
Created attachment 164586 [details] [review]
Bug 35659: (QA follow-up)
Comment 46 Matthias Meusburger 2024-04-10 09:12:44 UTC
Created attachment 164590 [details] [review]
Bug 35659: (QA follow-up)
Comment 47 Julian Maurice 2024-04-10 11:33:02 UTC
Created attachment 164602 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 48 Julian Maurice 2024-04-10 11:33:06 UTC
Created attachment 164603 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 49 Julian Maurice 2024-04-10 11:33:10 UTC
Created attachment 164604 [details] [review]
Bug 35659: (follow-up) Better handling of accented characters

If you try to harvest bibliographic records from a UNIMARC OAI
repository (using oai_dc data format) in a MARC21 Koha instance
and run the OAI harvester script in verbose mode, you may get
lines similar to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

When looking at the imported records' biblio details page in
the OPAC, most words containing accented characters will not
appear correctly.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT that transforms the UNIMARC OAI records
into MARC21 XML. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.

Test plan:

0) Without this patch, running the OAI harvesting script in
   verbose mode produces many warnings, and garbled characters
   appear in the OPAC biblio details page wherever accented
   characters are in use.

1) Apply this patch.

2) Re-run the OAI harvesting script in verbose + force mode
   (force mode is required to ignore record datestamps from
   previous runs):

   misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f

   This time there should be no warnings printed on your
   screen, and any characters with accents in the updated
   records should look OK in the OPAC.

Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 50 Julian Maurice 2024-04-10 11:33:13 UTC
Created attachment 164605 [details] [review]
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 51 Julian Maurice 2024-04-10 11:33:17 UTC
Created attachment 164606 [details] [review]
Bug 35659: (QA follow-up)

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 52 Pedro Amorim 2024-04-10 12:31:32 UTC
Created attachment 164618 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 53 Pedro Amorim 2024-04-10 12:31:56 UTC
Created attachment 164619 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 54 Pedro Amorim 2024-04-10 12:32:06 UTC
Created attachment 164620 [details] [review]
Bug 35659: (follow-up) Better handling of accented characters

If you try to harvest bibliographic records from a UNIMARC OAI
repository (using oai_dc data format) in a MARC21 Koha instance
and run the OAI harvester script in verbose mode, you may get
lines similar to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

When looking at the imported records' biblio details page in
the OPAC, most words containing accented characters will not
appear correctly.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT that transforms the UNIMARC OAI records
into MARC21 XML. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.

Test plan:

0) Without this patch, running the OAI harvesting script in
   verbose mode produces many warnings, and garbled characters
   appear in the OPAC biblio details page wherever accented
   characters are in use.

1) Apply this patch.

2) Re-run the OAI harvesting script in verbose + force mode
   (force mode is required to ignore record datestamps from
   previous runs):

   misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f

   This time there should be no warnings printed on your
   screen, and any characters with accents in the updated
   records should look OK in the OPAC.

Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 55 Pedro Amorim 2024-04-10 12:32:11 UTC
Created attachment 164621 [details] [review]
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 56 Pedro Amorim 2024-04-10 12:32:14 UTC
Created attachment 164622 [details] [review]
Bug 35659: (QA follow-up)

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 57 Jonathan Druart 2024-04-17 12:27:44 UTC
+        $dbh->do(q{DROP TABLE IF EXISTS `oaiservers`;});

You shouldn't drop the table but create it only if it does not exist.
Comment 58 Matthias Meusburger 2024-04-19 13:42:00 UTC
Created attachment 165199 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 59 Matthias Meusburger 2024-04-19 13:42:47 UTC
(In reply to Jonathan Druart from comment #57)
> +        $dbh->do(q{DROP TABLE IF EXISTS `oaiservers`;});
> 
> You shouldn't drop the table but create it only if it does not exist.

Thanks Joubu, this is fixed.
Comment 60 Tomás Cohen Arazi 2024-04-22 13:35:54 UTC
Some comments:

* Why is the `oaiservers` table name used instead of `oai_servers`? (same for oaiservers.{ps|tt})
* Why we sometimes call it `oai` and sometimes `oaipmh`?
* OaiServer should either be OAI::Server or OAIServer.
* Column names look old-school IMHO (`id`vs `oai_server_id`)
* Can we pick names that match biblio_metadata columns for example? Like `format` and `schema` instead of `recordtype`
* Is `datestamp` some OAI-PMH specific naming convention?
* `repository` varchar(255) NOT NULL COMMENT 'OAI repository'? Shouldn't this be a FK on the `oaiservers` table? This feels important as we are also adding 'record sources' on another bug, and we will eventually want to set rules for OAI-PMH imported records.
* Can the OAI harvester be thought as a background job in the mid/long term? In that case, it feels weird that Koha::OAI::Client::Harvester just prints things. It might be better to -for example- accumulate them using Koha::Object->add_message and have the caller decide what to do? (so harvest_oai.pl decides what to do) or it could just be a callback function for dealing with each step's output.
Comment 61 Victor Grousset/tuxayo 2024-04-26 04:15:38 UTC
> * Column names look old-school IMHO (`id`vs `oai_server_id`)

That's a definitely an issue:
https://wiki.koha-community.org/wiki/Coding_Guidelines#SQL7:_Primary_keys
«primary key name must be tablename_id (color_id for example) »

Also it seems for new columns we are now using snake_case:
https://wiki.koha-community.org/wiki/Database_updates#How_to_write_an_atomicupdate_file
«Example 2: Adding a new column»
Except for foreign keys like biblionumber


> * Why is the `oaiservers` table name used instead of `oai_servers`? (same for oaiservers.{ps|tt})

+1 for consistency with how new stuff is usually named.

---

Code:
> sub ServerSearch

This rule should also apply on new all new files outside C4:
https://wiki.koha-community.org/wiki/Coding_Guidelines#PERL9:_Subroutine_naming_conventions
«subroutines in the Koha namespace should be snake case»

---

String changes:

#: installer/data/mysql/en/mandatory/sample_notices.yml:letter:1:row:98:mul:4
msgid "Set: %s"

#: koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt:92
msgid "Set: "

+#: koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt:136
+#, fuzzy, c-format
+msgid "Set"

They need context for translators: https://wiki.koha-community.org/wiki/Internationalization,_plural_forms,_context,_and_more_RFC#More_on_context

And using tp() is an opportunity to deduplicate "Set: " and "Set"


+#: koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt:126
+#, c-format
+msgid "%s You searched for record %s %s You searched for %s %s "

This string should be split. Either via t() or <span>

---

I reattached the patches to fix the order that broke applying.
Comment 62 Victor Grousset/tuxayo 2024-04-26 04:15:52 UTC
Created attachment 165594 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 63 Victor Grousset/tuxayo 2024-04-26 04:15:57 UTC
Created attachment 165595 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 64 Victor Grousset/tuxayo 2024-04-26 04:16:01 UTC
Created attachment 165596 [details] [review]
Bug 35659: (follow-up) Better handling of accented characters

If you try to harvest bibliographic records from a UNIMARC OAI
repository (using oai_dc data format) in a MARC21 Koha instance
and run the OAI harvester script in verbose mode, you may get
lines similar to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

When looking at the imported records' biblio details page in
the OPAC, most words containing accented characters will not
appear correctly.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT that transforms the UNIMARC OAI records
into MARC21 XML. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.

Test plan:

0) Without this patch, running the OAI harvesting script in
   verbose mode produces many warnings, and garbled characters
   appear in the OPAC biblio details page wherever accented
   characters are in use.

1) Apply this patch.

2) Re-run the OAI harvesting script in verbose + force mode
   (force mode is required to ignore record datestamps from
   previous runs):

   misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f

   This time there should be no warnings printed on your
   screen, and any characters with accents in the updated
   records should look OK in the OPAC.

Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>
Signed-off-by: Michal Denar <black23@gmail.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 65 Victor Grousset/tuxayo 2024-04-26 04:16:05 UTC
Created attachment 165597 [details] [review]
Bug 35659: (follow-up) Make OAI repositories edition comply with CSRF

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Comment 66 Victor Grousset/tuxayo 2024-04-26 04:16:09 UTC
Created attachment 165598 [details] [review]
Bug 35659: (QA follow-up)

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>