Bug 40308

Summary: New Perl Class (Koha::ExternalContent::OCLC) for integrating with OCLC's APIs
Product: Koha Reporter: Andreas Roussos <a.roussos>
Component: MARC Authority data supportAssignee: Andreas Roussos <a.roussos>
Status: Needs Signoff --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low    
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: Medium patch Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on:    
Bug Blocks: 40309, 40310, 40311    
Attachments: Sample authority records (MARC21 flavour)
Sample authority records (UNIMARC flavour)
Bug 40308: add new Perl Class Koha::ExternalContent::OCLC
Bug 40308: add new System Preferences for OCLC Dewey API
Bug 40308: add a command-line tool that updates authorities en masse

Description Andreas Roussos 2025-07-06 21:01:58 UTC
Background information
----------------------
OCLC recently [1] started offering a Linked Data (LD) [2] API [3] that
allows looking up Dewey Decimal Classification (DDC) numbers [4] in order
to obtain unique URIs/URLs and Descriptions for a particular DDC (or range).
In addition to API searches, a smaller subset of the LD can also be browsed
online in a hierarchical manner [5].

This is just one from a large set of APIs offered by OCLC [6].

Reasoning behind this Bug report
--------------------------------
We believe that Koha users who include DDC numbers in their authority
cataloguing would benefit from an integration with the OCLC Dewey LD API
that would allow them to easily look up DDCs in exchange for LD URIs/URLs
and Descriptions.

To this end, we propose the addition of a new Perl Class that will offer
this functionality (Koha::ExternalContent::OCLC). Initially, the class
will offer functions for issuing API access tokens and for looking up DDC
URIs/URLs and Descriptions on the Dewey LD API. In the future, more
code be added to the class to allow for integration with more OCLC APIs.

This Bug report will also include a command-line tool that will enable
users to update their authorities with OCLC Linked Data en masse, useful
for authority records that already contain DDC information.

Three new authority Framework plugins that will use the functions
offered by the new Perl Class will be added as separate Bug reports.
These offer the ability to look up DDCs with a single mouse click
while inside the Authorities module cataloguing editor.

Finally, we propose the addition of four new System Preferences that
control the integration with the OCLC Dewey Linked Data API, namely:
- OCLCAPIWSKeyClientID: Stores the client ID component of your OCLC API WSKey
- OCLCAPIWSKeyClientSecret: Stores the client secret component of your OCLC API WSKey
- OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the lookup service
- OCLCDeweyLinkedDataLanguage: Sets the language of the DDC description to be fetched

References:
[1] https://www.oclc.org/en/news/releases/2024/20240815-uniform-resource-identifiers-for-ddc-numbers.html
[2] https://www.oclc.org/en/linked-data.html
[3] https://developer.api.oclc.org/dewey-ld
[4] https://www.oclc.org/content/dam/oclc/dewey/versions/print/intro.pdf
[5] https://entities.oclc.org/worldcat/ddc/
[6] https://www.oclc.org/developer/api/oclc-apis.en.html
Comment 1 Andreas Roussos 2025-07-06 23:42:59 UTC
Created attachment 183809 [details]
Sample authority records (MARC21 flavour)
Comment 2 Andreas Roussos 2025-07-06 23:43:25 UTC
Created attachment 183810 [details]
Sample authority records (UNIMARC flavour)
Comment 3 Andreas Roussos 2025-07-07 00:17:28 UTC
Created attachment 183811 [details] [review]
Bug 40308: add new Perl Class Koha::ExternalContent::OCLC

OCLC are offering a number of web APIs, this is an attempt
to collect utility functions in a new Perl Class.

At the moment, the Class contains the following functions,
that allow interaction with the OCLC Dewey Linked Data API:

- get_access_token
- get_ddc_uri
- get_ddc_url
- get_ddc_description

Test plan:

1) Inside a KTD instance (MARC flavour does not play a role),
   run `perldoc -o html Koha::ExternalContent::OCLC > OCLC.html`
   and view the documentation for this class (OCLC.html) in your
   web browser. Ensure everything makes sense ;-)
Comment 4 Andreas Roussos 2025-07-07 00:17:30 UTC
Created attachment 183812 [details] [review]
Bug 40308: add new System Preferences for OCLC Dewey API

This patch adds four new System Preferences that control the
integration with the OCLC Dewey Linked Data API, namely:

- OCLCAPIWSKeyClientID: Stores the client ID component of your
                        OCLC API WSKey
- OCLCAPIWSKeyClientSecret: Stores the client secret component
                            of your OCLC API WSKey
- OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the
                          lookup service
- OCLCDeweyLinkedDataLanguage: Sets the language of the DDC
                               description to be fetched

Test plan:

0) Launch either a MARC21 or a UNIMARC KTD container (MARC flavour does
   not play a role for this particular test plan), then apply this patch.

1) Run `reset_all` to force your KTD instance to load
   installer/data/mysql/mandatory/sysprefs.sql.

2) Then, visit Administration > System preferences > Web services and
   inspect the four new sysprefs under 'OCLC Dewey Linked Data API'.
   Look for typos in the descriptions, etc.

3) Then, run `updatedatabase` to test the atomic update.
   It should complete without warnings/errors.

4) Finally, visually inspect the output of `git log -p C4/UsageStats.pm`
   for errors. Notice that the System Preferences pertaining to the WSKey
   client ID and secret were *not* added for security reasons.
Comment 5 Andreas Roussos 2025-07-07 00:17:32 UTC
Created attachment 183813 [details] [review]
Bug 40308: add a command-line tool that updates authorities en masse

In Bugs 40309, 40310, and 40311 we will add three new Authority
Framework plugins that allow for easy DDC lookups using OCLC's
Dewey Linked Data API while cataloguing authorities in the Staff
Interface.

However, users may want to retroactively enrich their existing
authorities using a command-line tool that can modify authorities
en masse.

This patch adds misc/maintenance/fetch_oclc_dewey_linked_data.pl,
which is a Perl script that can accept a number of options in
order to facilitate selective operation on specific authority
types, or even authid ranges.

Test plan (NOTE: you need to have an organizational account with
OCLC and an active WebDewey subscription associated with it in
order to request a WSKey for accessing the Dewey Linked Data API):

[ I recommend working with two KTD instances in parallel. ]

0) First of all, run:
   ktd_proxy --start

1) Then, edit your KTD .env file so that:
   KOHA_MARC_FLAVOUR=marc21
   Then:
   ktd --proxy --name marc21 up
   Wait until it starts up.

   Then, in a different terminal, edit your KTD .env file so that:
   KOHA_MARC_FLAVOUR=unimarc
   Then:
   ktd --proxy --name unimarc up
   Wait until it starts up.

   You now have two KTDs running, one for each MARC flavour you
   want to test.

2) Get inside the MARC21 Koha container:

   ktd --name marc21 --shell

   Download the sample MARC21 authority records
   (sample_auth_records_marc21.mrc) attached to this
   Bug report (the file contains 15 records):
   wget -O sample_auth_records_marc21.mrc https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183809
   Then, import the authorities:
   misc/migration_tools/bulkmarcimport.pl -a --file sample_auth_records_marc21.mrc -v -c MARC21 -m ISO2709

   Repeat for the UNIMARC Koha container:

   ktd --name unimarc --shell
   Download the sample UNIMARC authority records
   (sample_auth_records_unimarc.mrc) attached to this
   Bug report (the file contains 12 records):
   wget -O sample_auth_records_unimarc.mrc https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183810
   Then, import the authorities:
   misc/migration_tools/bulkmarcimport.pl -a --file sample_auth_records_unimarc.mrc -v -c UNIMARC -m ISO2709

3) Once the sample authorities have been imported, you must fill in
   these two System Preferences (this is the tricky part, as, in order
   to request a WSKey you need to have an organizational account with
   OCLC and an active WebDewey subscription associated with it):
   - OCLCAPIWSKeyClientID
   - OCLCAPIWSKeyClientSecret

4) When the above System Preferences have been filled in, try running
   the new script (passing as many variations of the command-line
   options as you can). For example:

   cd misc/maintenance/
   ./fetch_oclc_dewey_linked_data.pl -h
   ./fetch_oclc_dewey_linked_data.pl --man | cat
   ./fetch_oclc_dewey_linked_data.pl -v
   ./fetch_oclc_dewey_linked_data.pl -v -l el
   ./fetch_oclc_dewey_linked_data.pl -v -a SNC
   ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955"
   ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -c
   ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -l it
   ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -l it -f
   ./fetch_oclc_dewey_linked_data.pl -v -l no -f
   ./fetch_oclc_dewey_linked_data.pl
   ...etc

   The output of the script for the above commands should resemble
   what is being depicted in the following asciinema screencast:

   https://asciinema.org/a/FKxWqNHLeZbxcgCBwYWZ4fod9

   If all goes well, you should have updated all authorities that
   have a DDC field with their matching Dewey Linked Data.