Bug 21359 - Add RDF support to OAI-PMH harvester
Summary: Add RDF support to OAI-PMH harvester
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Cook
QA Contact: Testopia
URL:
Keywords:
Depends on: 10662 18713 18586
Blocks:
  Show dependency treegraph
 
Reported: 2018-09-15 22:41 UTC by David Cook
Modified: 2018-09-16 21:56 UTC (History)
1 user (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 21359: Add RDF support to OAI-PMH harvester (29.01 KB, patch)
2018-09-16 18:25 UTC, David Cook
Details | Diff | Splinter Review
Empty RDF Triplestore Configuration File (756 bytes, text/plain)
2018-09-16 19:08 UTC, David Cook
Details
Bug 21359: Add RDF support to OAI-PMH harvester (28.92 KB, patch)
2018-09-16 19:27 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2018-09-15 22:41:33 UTC
The OAI-PMH harvester is set up to import only MARC records into Koha. The goal of this enhancement would be to allow RDF records to be imported into Koha as well.
Comment 1 David Cook 2018-09-16 18:25:52 UTC
Created attachment 78974 [details] [review]
Bug 21359: Add RDF support to OAI-PMH harvester

This patch allows the OAI-PMH harvester to process
RDFXML and place it in an external triplestore.

Currently, these RDFXML records must be linked to
MARCXML records via a matching OAI-PMH repository
and OAI-PMH identifier combination.
Comment 2 David Cook 2018-09-16 19:08:52 UTC
Created attachment 78993 [details]
Empty RDF Triplestore Configuration File

This file needs values to be filled in order to connect to a triplestore
Comment 3 David Cook 2018-09-16 19:27:32 UTC
Created attachment 78998 [details] [review]
Bug 21359: Add RDF support to OAI-PMH harvester

This patch allows the OAI-PMH harvester to process
RDFXML and place it in an external triplestore.

Currently, these RDFXML records must be linked to
MARCXML records via a matching OAI-PMH repository
and OAI-PMH identifier combination.
Comment 4 David Cook 2018-09-16 21:56:07 UTC
This setup/test plan is not a final draft as I don't have any suitable RDFXML sources via OAI-PMH to test at the moment, and it's not 100% clear what the best way to handle RDF in Koha is yet.


To setup Kohadevbox:

0. Apply patches (e.g. git bz apply 21359)
1. sudo apt-get install libpoe-perl libpoe-component-jobqueue-perl librdf-trine-perl librdf-query-perl 
2. In your browser, go to localhost:8081 and run web installer
3. cd /home/vagrant/kohaclone
4. sudo koha-shell kohadev -c "perl installer/data/mysql/updatedatabase.pl"
5a. Download OAI-PMH configuration file from https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78583 as oai.yml in /home/vagrant/kohaclone
5b. Download RDF Triplestore configuration file from
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78993 as triplestore.yaml in /home/vagrant/kohaclone
5c. The "url" values for "query" and "update" need to be filled in with a RDF triplestore (e.g. Apache Fuseki) in triplestore.yaml 
6. sudo vi /etc/koha/sites/kohadev/koha-conf.xml
Add the following before </config>:
<oai_pmh_harvester_config>/home/vagrant/kohaclone/oai.yml</oai_pmh_harvester_config>
<triplestore_config>/home/vagrant/kohaclone/triplestore.yaml</triplestore_config>
7. restart_all 
8. sudo KOHA_CONF=/etc/koha/sites/kohadev/koha-conf.xml PERL5LIB=/home/vagrant/kohaclone perl misc/harvesterd.pl --log-level DEBUG

To test:
1) Start OAI-PMH harvester daemon according to the above Kohadevbox instructions (koha-testing-docker can be used instead but will need some modifications)
2) Go to /cgi-bin/koha/tools/tools-home.pl
3) Click on "OAI-PMH harvester" (/cgi-bin/koha/tools/oai-pmh-harvester/dashboard.pl)
4) At the toolbar choose "New request"
5a) Give the request a "Name" (this is just used for Koha and not sent in the request), add "URL" for an OAI-PMH repository (e.g. http://<koha>/cgi-bin/koha/oai.pl), and fill in the OAI-PMH parameters as desired/required (an explanation of the protocol is at http://www.openarchives.org/OAI/openarchivesprotocol.html or see attached screenshot oai_request_example.jpg). The remaining values can be kept unchanged. 
5b) You'll need to specify a URI in "RDF type" for the type of RDF record you want to import. (Since RDF is essentially a container that contain many triples, you need to choose the type of the primary record you want to import)
6) Click "Test parameters" to make sure your inputs are all valid. 
7) Click "Save"
8) Click "Actions" and choose "Submit"
9) If the submission is successful, go to the "Submitted requests" tab
10) Click "Actions" and choose "Start"
11) Go to the "Import history" tab
12) Click "Refresh import history" to update the table and see your results. If nothing is coming up, you may need to change your request as the OAI-PMH repository may not have any results for that particular request.
13) You will need to check your triplestore directly to see if any content has been added