Bug 35659 - OAI Harvester
Summary: OAI Harvester
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Matthias Meusburger
QA Contact: Testopia
URL:
Keywords: no-sandbox
Depends on:
Blocks:
 
Reported: 2023-12-28 15:43 UTC by Matthias Meusburger
Modified: 2024-02-28 07:35 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 35659: OAI Harvester (99.13 KB, patch)
2023-12-28 16:00 UTC, Matthias Meusburger
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.11 KB, patch)
2023-12-28 16:00 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (99.56 KB, patch)
2024-01-10 09:35 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (102.98 KB, patch)
2024-02-07 15:11 UTC, Matthias Meusburger
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (102.95 KB, patch)
2024-02-09 09:20 UTC, Matthias Meusburger
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.15 KB, patch)
2024-02-10 20:09 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.00 KB, patch)
2024-02-10 20:09 UTC, Michal Denar
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.15 KB, patch)
2024-02-10 20:10 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.00 KB, patch)
2024-02-10 20:10 UTC, Michal Denar
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.19 KB, patch)
2024-02-10 20:12 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.04 KB, patch)
2024-02-10 20:12 UTC, Michal Denar
Details | Diff | Splinter Review
DO NOT PUSH: Bug 35659 DBIC Update (9.23 KB, patch)
2024-02-10 20:33 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 35659: OAI Harvester (103.08 KB, patch)
2024-02-10 20:33 UTC, Michal Denar
Details | Diff | Splinter Review
Screenshot - Accented characters in the OPAC (90.94 KB, image/png)
2024-02-28 07:32 UTC, Andreas Roussos
Details
Bug 35659: (follow-up) Better handling of accented characters (2.79 KB, patch)
2024-02-28 07:35 UTC, Andreas Roussos
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Meusburger 2023-12-28 15:43:23 UTC
Koha needs to be able to harvest records from OAI-PMH repositories.

This patch implements a cronjob-based OAI harvester, deliberately kept simple and consistent in its configuration with what we already have in Koha for other data sources, such as Z3950 and SRU.

It is sponsored by KohaLa, its goals and implementation have been previously discussed on the koha and koha-devel mailing-lists, as well as in a presentation at KohaCon'23 in Helsinki: "Let's add an OAI-PMH harvester in Koha" by Sonia Bouis.

Some of the code and ideas are based on a previous implementation of an OAI harvester by David Cook (See BZ 25905 and 10662).
Comment 1 Matthias Meusburger 2023-12-28 16:00:25 UTC
Created attachment 160356 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 2 Matthias Meusburger 2023-12-28 16:00:44 UTC
Created attachment 160357 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update
Comment 3 David Cook 2024-01-02 01:53:32 UTC
This is great news! Once my January holidays are done, I can look more at these patches!

At a glance, you will need to add unit tests for Koha::OAI::Client::Harvester before this can Pass QA (PERL17 in the coding guidelines), and that might require a bit of refactoring. (Missing unit tests are what ultimately bit me with bug 10662.)

I think that we're not allowing function declarations in .pl scripts anymore either (for admin/oaiservers.pl) so that might need to be refactored. That said, I can't find a reference to it in the coding guidelines and the patches pass the QA scripts, so I could be wrong there. 

But other than that the patches are looking pretty good at a glance I think.
Comment 4 David Cook 2024-01-02 01:54:12 UTC
I'm having some trouble running /kohadevbox/koha/t/db_dependent/Koha/OAIServers.t as part of the QA scripts, but it might be my install. I'm going to try to refresh my ktd...
Comment 5 David Cook 2024-01-02 02:10:49 UTC
I updated my ktd and now I'm getting failures when running the QA tools, although I've got the unit test working.

I notice there is a little typo in the pod in Koha/Import/Oaipmh/Authority.pm where it says "type" instead of "_type". Same with Koha/Import/Oaipmh/Biblio.pm and Koha/OaiServers.pm. 

Koha/OAI/Client/Harvester.pm is missing POD. 

I think the error for Koha/OaiServer.pm is false though. Not sure why that's getting a pod coverage error... 

I think the tidiness can also be ignored for the Schema files...

--

qa -c 2 -v 2
testing 2 commit(s) (applied to ac02c1d 'ad Bug 35405: Remove noise from MARC:')

Processing files before patches
The PO directory has not been found. There is a problem in your Koha installation. at /kohadevbox/koha/Koha/I18N.pm line 197.
|========================>| 27 / 27 (100.00%)
Processing files after patches
|========================>| 27 / 27 (100.00%)

 OK     Koha/Import/Oaipmh/Authorities.pm

 FAIL   Koha/Import/Oaipmh/Authority.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 FAIL   Koha/Import/Oaipmh/Biblio.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 OK     Koha/Import/Oaipmh/Biblios.pm

 FAIL   Koha/OAI/Client/Harvester.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 FAIL   Koha/OaiServer.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666

 OK     Koha/OaiServers.pm

 FAIL   Koha/Schema/Result/ImportOaipmhAuthority.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 33)

 FAIL   Koha/Schema/Result/ImportOaipmhBiblio.pm
   FAIL   pod coverage
                POD coverage was greater before, try perl -MPod::Coverage=PackageName -e666
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 28)

 FAIL   Koha/Schema/Result/Oaiserver.pm
   FAIL   pod coverage
                POD is missing for koha_object_class
                POD is missing for koha_objects_class
   WARN   tidiness
                The file is less tidy than before (bad/messy lines before: 0, now: 24)

 OK     admin/oaiservers.pl

 OK     installer/data/mysql/atomicupdate/Bug-35659.pl

 OK     installer/data/mysql/en/mandatory/sample_notices.yml

 OK     installer/data/mysql/kohastructure.sql

 OK     installer/data/mysql/mandatory/sysprefs.sql

 OK     koha-tmpl/intranet-tmpl/prog/en/includes/admin-menu.inc

 OK     koha-tmpl/intranet-tmpl/prog/en/includes/oai-admin-search.inc

 OK     koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc

 OK     koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt

 OK     koha-tmpl/intranet-tmpl/prog/en/modules/admin/oaiservers.tt

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl

 OK     koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 OK     misc/cronjobs/harvest_oai.pl

 OK     t/db_dependent/Koha/OAIServers.t


Processing additional checks

        * Commit title does not start with 'Bug XXXXX: ' - 44b002f8cc

Running tests (1)
        * Proving /kohadevbox/koha/t/db_dependent/Koha/OAIServers.t OK!
Comment 6 Katrin Fischer 2024-01-02 22:18:36 UTC
We are still missing supporters for this in the road map:
https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*
Comment 7 David Cook 2024-01-02 22:25:40 UTC
(In reply to Katrin Fischer from comment #6)
> We are still missing supporters for this in the road map:
> https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*

I've just added myself. Hopefully someone else can jump on board too.
Comment 8 Koha Team University Lyon 3 2024-01-06 11:54:46 UTC
(In reply to Katrin Fischer from comment #6)
> We are still missing supporters for this in the road map:
> https://wiki.koha-community.org/wiki/Road_map_24.05 *hint*

Add myself as supporter :)
Comment 9 Matthias Meusburger 2024-01-10 09:35:36 UTC
Created attachment 160744 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 10 Matthias Meusburger 2024-01-10 09:45:34 UTC
Hi David!

The typo in Koha/Import/Oaipmh/Authority.pm, Koha/Import/Oaipmh/Biblio.pm and Koha/OaiServers.pm has been fixed.

POD has been added to Koha/OAI/Client/Harvester.pm

> I think the error for Koha/OaiServer.pm is false though. Not sure why that's getting a pod coverage error... 

About those, ashimema told me on IRC: 

15:07	ashimema	if they're new classes you can safely ignore that..
15:07		there's a bug in the POD detection for new clases

> I think the tidiness can also be ignored for the Schema files...

That's what I thought too, since they are auto-generated.
Comment 11 David Cook 2024-01-22 22:44:53 UTC
I'm in my last week of holiday mode, so I should be more available for this in the coming weeks.

Thanks Matthias for looking at that. I'll take another look soon at the QA tools output another time.

It looks like Koha::OAI::Client::Harvester is still missing unit tests though. I think that's going to be a QA blocker.
Comment 12 David Cook 2024-01-22 22:45:42 UTC
Comment on attachment 160744 [details] [review]
Bug 35659: OAI Harvester

Review of attachment 160744 [details] [review]:
-----------------------------------------------------------------

::: Koha/OAI/Client/Harvester.pm
@@ +127,5 @@
> +        printlog("ListMetadataFormats failed");
> +    };
> +
> +    #my $response = $h->Identify;
> +    #print "Repository supports protocol version ", $response->version, "\n";

Looks like some debugging code that needs to be removed here
Comment 13 David Cook 2024-01-22 22:58:37 UTC
(In reply to David Cook from comment #11)
> It looks like Koha::OAI::Client::Harvester is still missing unit tests
> though. I think that's going to be a QA blocker.

At a minimum, I think the code handling "$response->next" should be in separate unit tested functions. 

(In theory, the other earlier code within the "new" constructor could probably be moved into an "init" function as well. But that is harder to test since it interacts with a real website. We could create a fake website but I think that would be a bit above and beyond.)
Comment 14 Matthias Meusburger 2024-02-07 15:11:05 UTC
Created attachment 161813 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 15 Matthias Meusburger 2024-02-07 15:12:21 UTC
(In reply to David Cook from comment #13)
> (In reply to David Cook from comment #11)
> > It looks like Koha::OAI::Client::Harvester is still missing unit tests
> > though. I think that's going to be a QA blocker.
> 
> At a minimum, I think the code handling "$response->next" should be in
> separate unit tested functions. 
> 

Hi! I've added unit tests for record processing.
Comment 16 David Cook 2024-02-07 22:38:10 UTC
(In reply to Matthias Meusburger from comment #15)
> Hi! I've added unit tests for record processing.

Thanks, Matthias! I've taken a quick glance at the unit tests and they're looking good! 

I've got a bit on my plate at the moment, but I'll come back to look at this soon! 

I think it would be good to find someone else to Sign Off on it, so that I can QA it.
Comment 17 Michal Denar 2024-02-08 20:38:27 UTC
Hi,
I get an error when I try to apply the patch:

Bug 35659 - OAI Harvester

160357 - DO NOT PUSH: Bug 35659 DBIC Update
161813 - Bug 35659: OAI Harvester

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: DO NOT PUSH: Bug 35659 DBIC Update
Applying: Bug 35659: OAI Harvester
Using index info to reconstruct a base tree...
M	installer/data/mysql/mandatory/sysprefs.sql
M	installer/data/mysql/mandatory/userpermissions.sql
M	koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc
M	koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/intranet-tmpl/prog/en/modules/admin/admin-home.tt
Auto-merging koha-tmpl/intranet-tmpl/prog/en/includes/permissions.inc
Auto-merging installer/data/mysql/mandatory/userpermissions.sql
Auto-merging installer/data/mysql/mandatory/sysprefs.sql
CONFLICT (content): Merge conflict in installer/data/mysql/mandatory/sysprefs.sql
error: Failed to merge in the changes.
Comment 18 Matthias Meusburger 2024-02-09 09:20:44 UTC
Created attachment 161960 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Comment 19 Matthias Meusburger 2024-02-09 09:22:17 UTC
(In reply to Michal Denar from comment #17)
> Hi,
> I get an error when I try to apply the patch:
> 

Hi Michal, sorry about that. The patch has been rebased to master, it should be ok now.
Comment 20 Michal Denar 2024-02-09 21:31:42 UTC
Hi,
this patch works for me. I don't know why, but I can't get it to unsign the signed patch and switch to SO. I've tried several times, but without success. Harvester works and it is possible to switch to signed off.
Comment 21 Michal Denar 2024-02-09 21:34:45 UTC
I get error: "Failed to update bug 35659, status=200"
BZ login credentials are correct.
Comment 22 Michal Denar 2024-02-10 20:09:00 UTC
Created attachment 162004 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: John Doe <you@example.com>
Comment 23 Michal Denar 2024-02-10 20:09:03 UTC
Created attachment 162005 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Comment 24 Michal Denar 2024-02-10 20:10:00 UTC
Created attachment 162006 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: John Doe <you@example.com>
Comment 25 Michal Denar 2024-02-10 20:10:04 UTC
Created attachment 162007 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Comment 26 Michal Denar 2024-02-10 20:12:34 UTC
Created attachment 162008 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal <black232gmail.com>
Comment 27 Michal Denar 2024-02-10 20:12:40 UTC
Created attachment 162009 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Signed-off-by: Michal <black232gmail.com>
Comment 28 Michal Denar 2024-02-10 20:33:32 UTC
Created attachment 162010 [details] [review]
DO NOT PUSH: Bug 35659 DBIC Update

Signed-off-by: Michal <black23@gmail.com>
Comment 29 Michal Denar 2024-02-10 20:33:35 UTC
Created attachment 162011 [details] [review]
Bug 35659: OAI Harvester

Test plan:

 - Run updatedatabase.pl

 - Apply the DBIC update patch

 - Setup an OAI repository in Administration -> OAI repositories (admin/oaiservers.pl)

   If the repository dataformat is in oai_dc, you can use the sample XSLT files:
   For Unimarc:
   OAIDCtoUNIMARCXML.xsl for bibliographic records
   AuthOAIDCtoUNIMARCXML.xsl for authorities

   For Marc21:
   OAIDCtoMARC21XML.xsl for bibliographic records
   AuthOAIDCtoMARC21XML.xsl for authorities

   If the repository dataformat is in marc-xml, you can use
   OAIMarcxml2KohaMarcxml.xsl for both Unimarc and Marc21.

 - Some repositories to test with:

   Bibliographic records, oai_dc:
   Endpoint: http://staroai.theses.fr/OAIHandler
   Set: ddc:260
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIDCtoMARC21XML.xsl

   Authorities, oai_dc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoUNIMARCXML.xsl
   or
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/AuthOAIDCtoMARC21XML.xsl

   Authorities, marc-xml, Unimarc:
   Endpoint: https://www.idref.fr/OAI/oai.jsp
   Set: a
   XSLT: <path_to_src>/koha-tmpl/intranet-tmpl/prog/en/xslt/OAIMarcxml2KohaMarcxml.xsl

 - Note: marc-xml harvesting will not work with HTTP::OAI::Harvester version 3.xx
   Version 4.xx is needed.
   See Bug 17704 for updating Koha default version to 4.xx

 - If you want to receive a report email, fill the OAI-PMH:HarvestEmailReport
   syspref. The model used (OAI_HARVEST_REPORT) can be edited in tools -> Notices and slips.

 - start a harvest with the following command:
   misc/cronjobs/harvest_oai.pl

    This script starts an OAI Harvest

    This script has the following parameters :
        -h --help: this message
        -v --verbose (enable output to stdout)
        -r --repository: id of the OAI repository
        -d --days: number of days to harvest from (optional)
        -l --list: list the OAI repositories
        -f --force: force harvesting (ignore records datestamps) (optional)

    Example: misc/cronjobs/harvest_oai.pl -r 1 -d 10 -v -f

 - Check the logs in the Log Viewer (tools/viewlog.pl)
   (CronjobLog syspref must be activated)

Sponsored-by: KohaLa
Signed-off-by: John Doe <you@example.com>
Signed-off-by: Michal <black232gmail.com>
Signed-off-by: John Doe <you@example.com>
Comment 30 Andreas Roussos 2024-02-28 07:32:19 UTC
Created attachment 162527 [details]
Screenshot - Accented characters in the OPAC
Comment 31 Andreas Roussos 2024-02-28 07:34:07 UTC
Just wanted to point out that I tried to harvest bibliographic
records from the first repository suggested in the Test plan
(http://staroai.theses.fr/OAIHandler, oai_dc data format) in a
MARC21 Koha instance (therefore using OAIDCtoMARC21XML.xsl to
transform the records prior to importing), and when I ran the
OAI harvester script in verbose mode I got many lines similar
to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

The script did actually finish and managed to add 91 records,
but I noticed that in the biblio details page in the OPAC some
(not all) words containing accented characters did not appear
correctly. I have uploaded a screenshot that illustrates this.

Upon further investigation, it looks like the script *does*
manage to populate database fields such as biblio.title and
biblio.abstract with the correctly accented characters, but
some MARC fields (such as 245 - Title, and 6xx - Subjects)
in the MARCXML representation in biblio_metadata.metadata
still contain weird characters wherever accents exist.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.
Comment 32 Andreas Roussos 2024-02-28 07:35:14 UTC
Created attachment 162528 [details] [review]
Bug 35659: (follow-up) Better handling of accented characters

If you try to harvest bibliographic records from a UNIMARC OAI
repository (using oai_dc data format) in a MARC21 Koha instance
and run the OAI harvester script in verbose mode, you may get
lines similar to the following in the output:

no mapping found for [0xC9] at position 0 in Économie politique g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.
no mapping found for [0xC9] at position 0 in Église et société g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308.

When looking at the imported records' biblio details page in
the OPAC, most words containing accented characters will not
appear correctly.

The fix is to apply Franck Theeten's solution from Bug 16488
(https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16488#c24)
and modify the value of the MARC leader's 10th character
to 'a' in the XSLT that transforms the UNIMARC OAI records
into MARC21 XML. Then, the accented characters get imported
properly and the records appear correctly in the OPAC.

Test plan:

0) Without this patch, running the OAI harvesting script in
   verbose mode produces many warnings, and garbled characters
   appear in the OPAC biblio details page wherever accented
   characters are in use.

1) Apply this patch.

2) Re-run the OAI harvesting script in verbose + force mode
   (force mode is required to ignore record datestamps from
   previous runs):

   misc/cronjobs/harvest_oai.pl -v -r <OAI_REPO_ID> -f

   This time there should be no warnings printed on your
   screen, and any characters with accents in the updated
   records should look OK in the OPAC.

Thanks-to: Franck Theeten <franck.theeten@africamuseum.be>