Background information ---------------------- OCLC recently [1] started offering a Linked Data (LD) [2] API [3] that allows looking up Dewey Decimal Classification (DDC) numbers [4] in order to obtain unique URIs/URLs and Descriptions for a particular DDC (or range). In addition to API searches, a smaller subset of the LD can also be browsed online in a hierarchical manner [5]. This is just one from a large set of APIs offered by OCLC [6]. Reasoning behind this Bug report -------------------------------- We believe that Koha users who include DDC numbers in their authority cataloguing would benefit from an integration with the OCLC Dewey LD API that would allow them to easily look up DDCs in exchange for LD URIs/URLs and Descriptions. To this end, we propose the addition of a new Perl Class that will offer this functionality (Koha::ExternalContent::OCLC). Initially, the class will offer functions for issuing API access tokens and for looking up DDC URIs/URLs and Descriptions on the Dewey LD API. In the future, more code be added to the class to allow for integration with more OCLC APIs. This Bug report will also include a command-line tool that will enable users to update their authorities with OCLC Linked Data en masse, useful for authority records that already contain DDC information. Three new authority Framework plugins that will use the functions offered by the new Perl Class will be added as separate Bug reports. These offer the ability to look up DDCs with a single mouse click while inside the Authorities module cataloguing editor. Finally, we propose the addition of four new System Preferences that control the integration with the OCLC Dewey Linked Data API, namely: - OCLCAPIWSKeyClientID: Stores the client ID component of your OCLC API WSKey - OCLCAPIWSKeyClientSecret: Stores the client secret component of your OCLC API WSKey - OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the lookup service - OCLCDeweyLinkedDataLanguage: Sets the language of the DDC description to be fetched References: [1] https://www.oclc.org/en/news/releases/2024/20240815-uniform-resource-identifiers-for-ddc-numbers.html [2] https://www.oclc.org/en/linked-data.html [3] https://developer.api.oclc.org/dewey-ld [4] https://www.oclc.org/content/dam/oclc/dewey/versions/print/intro.pdf [5] https://entities.oclc.org/worldcat/ddc/ [6] https://www.oclc.org/developer/api/oclc-apis.en.html
Created attachment 183809 [details] Sample authority records (MARC21 flavour)
Created attachment 183810 [details] Sample authority records (UNIMARC flavour)
Created attachment 183811 [details] [review] Bug 40308: add new Perl Class Koha::ExternalContent::OCLC OCLC are offering a number of web APIs, this is an attempt to collect utility functions in a new Perl Class. At the moment, the Class contains the following functions, that allow interaction with the OCLC Dewey Linked Data API: - get_access_token - get_ddc_uri - get_ddc_url - get_ddc_description Test plan: 1) Inside a KTD instance (MARC flavour does not play a role), run `perldoc -o html Koha::ExternalContent::OCLC > OCLC.html` and view the documentation for this class (OCLC.html) in your web browser. Ensure everything makes sense ;-)
Created attachment 183812 [details] [review] Bug 40308: add new System Preferences for OCLC Dewey API This patch adds four new System Preferences that control the integration with the OCLC Dewey Linked Data API, namely: - OCLCAPIWSKeyClientID: Stores the client ID component of your OCLC API WSKey - OCLCAPIWSKeyClientSecret: Stores the client secret component of your OCLC API WSKey - OCLCDeweyLinkedDataAPI: Basically an On/Off "switch" for the lookup service - OCLCDeweyLinkedDataLanguage: Sets the language of the DDC description to be fetched Test plan: 0) Launch either a MARC21 or a UNIMARC KTD container (MARC flavour does not play a role for this particular test plan), then apply this patch. 1) Run `reset_all` to force your KTD instance to load installer/data/mysql/mandatory/sysprefs.sql. 2) Then, visit Administration > System preferences > Web services and inspect the four new sysprefs under 'OCLC Dewey Linked Data API'. Look for typos in the descriptions, etc. 3) Then, run `updatedatabase` to test the atomic update. It should complete without warnings/errors. 4) Finally, visually inspect the output of `git log -p C4/UsageStats.pm` for errors. Notice that the System Preferences pertaining to the WSKey client ID and secret were *not* added for security reasons.
Created attachment 183813 [details] [review] Bug 40308: add a command-line tool that updates authorities en masse In Bugs 40309, 40310, and 40311 we will add three new Authority Framework plugins that allow for easy DDC lookups using OCLC's Dewey Linked Data API while cataloguing authorities in the Staff Interface. However, users may want to retroactively enrich their existing authorities using a command-line tool that can modify authorities en masse. This patch adds misc/maintenance/fetch_oclc_dewey_linked_data.pl, which is a Perl script that can accept a number of options in order to facilitate selective operation on specific authority types, or even authid ranges. Test plan (NOTE: you need to have an organizational account with OCLC and an active WebDewey subscription associated with it in order to request a WSKey for accessing the Dewey Linked Data API): [ I recommend working with two KTD instances in parallel. ] 0) First of all, run: ktd_proxy --start 1) Then, edit your KTD .env file so that: KOHA_MARC_FLAVOUR=marc21 Then: ktd --proxy --name marc21 up Wait until it starts up. Then, in a different terminal, edit your KTD .env file so that: KOHA_MARC_FLAVOUR=unimarc Then: ktd --proxy --name unimarc up Wait until it starts up. You now have two KTDs running, one for each MARC flavour you want to test. 2) Get inside the MARC21 Koha container: ktd --name marc21 --shell Download the sample MARC21 authority records (sample_auth_records_marc21.mrc) attached to this Bug report (the file contains 15 records): wget -O sample_auth_records_marc21.mrc https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183809 Then, import the authorities: misc/migration_tools/bulkmarcimport.pl -a --file sample_auth_records_marc21.mrc -v -c MARC21 -m ISO2709 Repeat for the UNIMARC Koha container: ktd --name unimarc --shell Download the sample UNIMARC authority records (sample_auth_records_unimarc.mrc) attached to this Bug report (the file contains 12 records): wget -O sample_auth_records_unimarc.mrc https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=183810 Then, import the authorities: misc/migration_tools/bulkmarcimport.pl -a --file sample_auth_records_unimarc.mrc -v -c UNIMARC -m ISO2709 3) Once the sample authorities have been imported, you must fill in these two System Preferences (this is the tricky part, as, in order to request a WSKey you need to have an organizational account with OCLC and an active WebDewey subscription associated with it): - OCLCAPIWSKeyClientID - OCLCAPIWSKeyClientSecret 4) When the above System Preferences have been filled in, try running the new script (passing as many variations of the command-line options as you can). For example: cd misc/maintenance/ ./fetch_oclc_dewey_linked_data.pl -h ./fetch_oclc_dewey_linked_data.pl --man | cat ./fetch_oclc_dewey_linked_data.pl -v ./fetch_oclc_dewey_linked_data.pl -v -l el ./fetch_oclc_dewey_linked_data.pl -v -a SNC ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -c ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -l it ./fetch_oclc_dewey_linked_data.pl -v -a SNC -w "authid = 5955" -l it -f ./fetch_oclc_dewey_linked_data.pl -v -l no -f ./fetch_oclc_dewey_linked_data.pl ...etc The output of the script for the above commands should resemble what is being depicted in the following asciinema screencast: https://asciinema.org/a/FKxWqNHLeZbxcgCBwYWZ4fod9 If all goes well, you should have updated all authorities that have a DDC field with their matching Dewey Linked Data.