Bug 21359

Created attachment 78974 [details] [review]
Bug 21359: Add RDF support to OAI-PMH harvester

This patch allows the OAI-PMH harvester to process
RDFXML and place it in an external triplestore.

Currently, these RDFXML records must be linked to
MARCXML records via a matching OAI-PMH repository
and OAI-PMH identifier combination.

Created attachment 78993 [details]
Empty RDF Triplestore Configuration File

This file needs values to be filled in order to connect to a triplestore

Created attachment 78998 [details] [review]
Bug 21359: Add RDF support to OAI-PMH harvester

This patch allows the OAI-PMH harvester to process
RDFXML and place it in an external triplestore.

Currently, these RDFXML records must be linked to
MARCXML records via a matching OAI-PMH repository
and OAI-PMH identifier combination.

This setup/test plan is not a final draft as I don't have any suitable RDFXML sources via OAI-PMH to test at the moment, and it's not 100% clear what the best way to handle RDF in Koha is yet.


To setup Kohadevbox:

0. Apply patches (e.g. git bz apply 21359)
1. sudo apt-get install libpoe-perl libpoe-component-jobqueue-perl librdf-trine-perl librdf-query-perl 
2. In your browser, go to localhost:8081 and run web installer
3. cd /home/vagrant/kohaclone
4. sudo koha-shell kohadev -c "perl installer/data/mysql/updatedatabase.pl"
5a. Download OAI-PMH configuration file from https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78583 as oai.yml in /home/vagrant/kohaclone
5b. Download RDF Triplestore configuration file from
https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=78993 as triplestore.yaml in /home/vagrant/kohaclone
5c. The "url" values for "query" and "update" need to be filled in with a RDF triplestore (e.g. Apache Fuseki) in triplestore.yaml 
6. sudo vi /etc/koha/sites/kohadev/koha-conf.xml
Add the following before </config>:
<oai_pmh_harvester_config>/home/vagrant/kohaclone/oai.yml</oai_pmh_harvester_config>
<triplestore_config>/home/vagrant/kohaclone/triplestore.yaml</triplestore_config>
7. restart_all 
8. sudo KOHA_CONF=/etc/koha/sites/kohadev/koha-conf.xml PERL5LIB=/home/vagrant/kohaclone perl misc/harvesterd.pl --log-level DEBUG

To test:
1) Start OAI-PMH harvester daemon according to the above Kohadevbox instructions (koha-testing-docker can be used instead but will need some modifications)
2) Go to /cgi-bin/koha/tools/tools-home.pl
3) Click on "OAI-PMH harvester" (/cgi-bin/koha/tools/oai-pmh-harvester/dashboard.pl)
4) At the toolbar choose "New request"
5a) Give the request a "Name" (this is just used for Koha and not sent in the request), add "URL" for an OAI-PMH repository (e.g. http://<koha>/cgi-bin/koha/oai.pl), and fill in the OAI-PMH parameters as desired/required (an explanation of the protocol is at http://www.openarchives.org/OAI/openarchivesprotocol.html or see attached screenshot oai_request_example.jpg). The remaining values can be kept unchanged. 
5b) You'll need to specify a URI in "RDF type" for the type of RDF record you want to import. (Since RDF is essentially a container that contain many triples, you need to choose the type of the primary record you want to import)
6) Click "Test parameters" to make sure your inputs are all valid. 
7) Click "Save"
8) Click "Actions" and choose "Submit"
9) If the submission is successful, go to the "Submitted requests" tab
10) Click "Actions" and choose "Start"
11) Go to the "Import history" tab
12) Click "Refresh import history" to update the table and see your results. If nothing is coming up, you may need to change your request as the OAI-PMH repository may not have any results for that particular request.
13) You will need to check your triplestore directly to see if any content has been added

My thoughts on this one have changed a lot over the years due to research and experience.

Really we need to add RDF support to Koha before we do anything else.