Bug 18585 - Connect to RDF triplestore
Summary: Connect to RDF triplestore
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Cook
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 10662 10787 10788 18713
  Show dependency treegraph
 
Reported: 2017-05-12 02:24 UTC by David Cook
Modified: 2018-04-10 00:03 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 18585 - Connect to RDF triplestore (7.21 KB, patch)
2017-05-12 05:04 UTC, David Cook
Details | Diff | Splinter Review
Bug 18585 - Connect to RDF triplestore (add dependency) (816 bytes, patch)
2017-05-12 06:54 UTC, David Cook
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 18585 - Connect to RDF triplestore (7.28 KB, patch)
2017-05-12 07:27 UTC, Josef Moravec
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 18585 - Connect to RDF triplestore (add dependency) (885 bytes, patch)
2017-05-12 07:27 UTC, Josef Moravec
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 18585 - Followup - fix pod (586 bytes, patch)
2017-05-12 07:27 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 18585 - Connect to RDF triplestore (9.87 KB, patch)
2017-06-13 06:29 UTC, David Cook
Details | Diff | Splinter Review
Bug 18585 - Connect to RDF triplestore (10.16 KB, patch)
2017-06-13 06:30 UTC, David Cook
Details | Diff | Splinter Review
Bug 18585 - Connect to RDF triplestore (10.15 KB, patch)
2017-12-08 08:38 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 18585 - Connect to RDF triplestore (10.15 KB, patch)
2018-01-29 01:19 UTC, David Cook
Details | Diff | Splinter Review
Bug 18585 - Connect to RDF triplestore (10.15 KB, patch)
2018-04-10 00:02 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2017-05-12 02:24:26 UTC
In order to store RDF data, we need to be able to connect to a triplestore. This enhancement simply allows us to connect to a separate triplestore.
Comment 1 David Cook 2017-05-12 05:04:12 UTC
Created attachment 63393 [details] [review]
Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context, which
returns a RDF::Trine::Model object if triplestore.yaml is
properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.
Comment 2 David Cook 2017-05-12 06:54:26 UTC
Created attachment 63403 [details] [review]
Bug 18585 - Connect to RDF triplestore (add dependency)

Add RDF::Trine to the list of Perl dependencies
Comment 3 Josef Moravec 2017-05-12 07:27:14 UTC
Created attachment 63411 [details] [review]
[SIGNED-OFF] Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context, which
returns a RDF::Trine::Model object if triplestore.yaml is
properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 4 Josef Moravec 2017-05-12 07:27:18 UTC
Created attachment 63412 [details] [review]
[SIGNED-OFF] Bug 18585 - Connect to RDF triplestore (add dependency)

Add RDF::Trine to the list of Perl dependencies

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 5 Josef Moravec 2017-05-12 07:27:22 UTC
Created attachment 63413 [details] [review]
[SIGNED-OFF] Bug 18585 - Followup - fix pod

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 6 Magnus Enger 2017-05-30 07:19:35 UTC
At least the Fuseki triplestore has different endpoints for reading and writing data. With Fuseki as the triplestore, this bug only adds a read RDF::Trine::Model object. Would it be possible to do a followup that would aloow us to create both read and write objects? (As two separate objects.)
Comment 7 David Cook 2017-06-01 05:50:08 UTC
(In reply to Magnus Enger from comment #6)
> At least the Fuseki triplestore has different endpoints for reading and
> writing data. With Fuseki as the triplestore, this bug only adds a read
> RDF::Trine::Model object. Would it be possible to do a followup that would
> aloow us to create both read and write objects? (As two separate objects.)

Hi Magnus,

I'm not sure why you think it adds a read-only object. I can 100% assure you that the RDF::Trine::Model object is both read and write. The URL you provide is to the dataset (e.g. http://semantikoha.libriotech.no:3030/david-koha-test). From there, RDF::Trine::Model will build the "update" and "query" URLs dynamically for reading and writing using the SPARQL 1.1 standard. (In fact, this is why RDF::Trine doesn't work with older versions of Virtuoso, since Virtuoso before 6.1.7 or so didn't comply with SPARQL 1.1.)
Comment 8 David Cook 2017-06-01 05:59:49 UTC
(In reply to David Cook from comment #7)
> (In reply to Magnus Enger from comment #6)
> > At least the Fuseki triplestore has different endpoints for reading and
> > writing data. With Fuseki as the triplestore, this bug only adds a read
> > RDF::Trine::Model object. Would it be possible to do a followup that would
> > aloow us to create both read and write objects? (As two separate objects.)
> 
> Hi Magnus,
> 
> I'm not sure why you think it adds a read-only object. I can 100% assure you
> that the RDF::Trine::Model object is both read and write. The URL you
> provide is to the dataset (e.g.
> http://semantikoha.libriotech.no:3030/david-koha-test). From there,
> RDF::Trine::Model will build the "update" and "query" URLs dynamically for
> reading and writing using the SPARQL 1.1 standard. (In fact, this is why
> RDF::Trine doesn't work with older versions of Virtuoso, since Virtuoso
> before 6.1.7 or so didn't comply with SPARQL 1.1.)

Actually, as I read through the code at http://cpansearch.perl.org/src/GWILLIAMS/RDF-Trine-1.016/lib/RDF/Trine/Store/SPARQL.pm, it looks like they don't dynamically change the URL. Rather, they just use the "update" and "query" parameters depending on the operation specified, but it's all using the same URL which as I mentioned above is just to the dataset.
Comment 9 David Cook 2017-06-01 06:04:48 UTC
Looking at https://jena.apache.org/documentation/serving_data/, it looks like "/data" is a read/write endpoint, and after visiting http://semantikoha.libriotech.no:3030/david-koha-test/ and http://semantikoha.libriotech.no:3030/david-koha-test/data, it seems that both map to the same place.

In which case... maybe it is worthwhile having separate read and write endpoints in the configuration file...
Comment 10 David Cook 2017-06-01 06:05:53 UTC
However, RDF::Trine::Store::Memory is read/write and I imagine RDF::Trine::Store::DBI is as well. Maybe worth looking at a few triplestores to see what the norm is.
Comment 11 David Cook 2017-06-01 06:08:44 UTC
Virtuoso still seems to have the wrong parameters but looks like it has a read/write endpoint (https://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VOSSparqlProtocol#HTTP Request Methods).
Comment 12 David Cook 2017-06-01 06:14:28 UTC
More info about Virtuoso even though we can't use it anyway with RDF::Trine (https://virtuoso.openlinksw.com/dataspace/doc/dav/wiki/Main/VirtTipsAndTricksGuideSPARQLEndpoints).
Comment 13 Magnus Enger 2017-06-01 07:09:17 UTC
Thanks for clarifying, David! I think the problem was that I put http://example.com/david-koha-test/query in the config, and that made the object read only. 

On the other hand, having separate read and read/write objects could perhaps make some sense, like using the read only object when you know you are only going to display data, like in the OPAC. (Especially for my librarian-configurable queries, to protect against people doing stupid/evil things :-)
Comment 14 David Cook 2017-06-02 01:52:52 UTC
(In reply to Magnus Enger from comment #13)
> Thanks for clarifying, David! I think the problem was that I put
> http://example.com/david-koha-test/query in the config, and that made the
> object read only. 
> 
> On the other hand, having separate read and read/write objects could perhaps
> make some sense, like using the read only object when you know you are only
> going to display data, like in the OPAC. (Especially for my
> librarian-configurable queries, to protect against people doing stupid/evil
> things :-)

That's very true, and that's something that I had been thinking about before and totally forgot about!

I think using a read-only endpoint for the librarian-configurable queries is a great idea, and closes off a security hole. 

Does Fuseki allow you to configure permissions for user accounts as well? So even if you do use a read/write endpoint, only certain users can insert or delete triples?
Comment 15 David Cook 2017-06-02 02:58:52 UTC
I'm thinking a little bit about what this would look like... 

I'm thinking maybe:

Code:
C4::Context::triplestore($model_name)

Config:

model_name: query
module:
url: 
username:
password:
realm:

model_name: update
module:
url: 
username:
password:
realm:

model_name: custom
module:
url: 
username:
password:
realm:
#NOTE: I doubt we'd ever use custom... but the flexibility would be there. Maybe there will be a case where in the future we want to be able to connect to multiple triplestores for some reason. Who knows?

--

This would actually work well with how RDF::Trine::Store::DBI is set up since it uses "model name" in its constructor. 
http://search.cpan.org/dist/RDF-Trine/lib/RDF/Trine/Store/DBI.pm
Comment 16 David Cook 2017-06-02 07:14:23 UTC
I ran out of time today and it's Friday, but I'll look at posting a revised patch on Monday. 

The patch needed to be rebased anyway because of some changes to koha-conf.xml I believe.
Comment 17 David Cook 2017-06-13 06:29:40 UTC
Created attachment 64252 [details] [review]
Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context. It takes
a model name as a parameter and returns a RDF::Trine::Model object
if triplestore.yaml is properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too,
or any other RDF::Trine::Store::* backend.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.
Comment 18 David Cook 2017-06-13 06:30:43 UTC
Created attachment 64253 [details] [review]
Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context. It takes
a model name as a parameter and returns a RDF::Trine::Model object
if triplestore.yaml is properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too,
or any other RDF::Trine::Store::* backend.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.
Comment 19 David Cook 2017-06-13 06:35:09 UTC
Magnus, how does that look now?

That's probably a safer way of doing things. That way, people could inject malicious or flawed SPARQL into a "read" query and do zero damage to the triplestore.

In the long run, having separate endpoints also might make it easier to manage load on the triplestore. A person could update the canonical triplestore and then send queries to a mirror which could be specified in the configuration file.
Comment 20 David Cook 2017-06-16 04:51:23 UTC
Updating my existing code now and this is turning out to be interesting...

In a test, I was using a "fake" model name, but my test failed, because one of the unrelated methods is hard-coded to use the "query" model.

I changed the model name in the test and it worked fine, but just something to keep in mind.
Comment 21 Katrin Fischer 2017-10-15 13:37:46 UTC
Assigning to David as he is the author of the patch. Patch still applies cleanly.
Comment 22 Josef Moravec 2017-12-08 08:38:57 UTC
Created attachment 69632 [details] [review]
Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context. It takes
a model name as a parameter and returns a RDF::Trine::Model object
if triplestore.yaml is properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too,
or any other RDF::Trine::Store::* backend.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 23 David Cook 2018-01-29 01:12:02 UTC
Looks like this one doesn't apply anymore... I'll rebase it in a minute
Comment 24 David Cook 2018-01-29 01:19:14 UTC
Created attachment 71016 [details] [review]
Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context. It takes
a model name as a parameter and returns a RDF::Trine::Model object
if triplestore.yaml is properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too,
or any other RDF::Trine::Store::* backend.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.
Comment 25 Charles Farmer 2018-03-30 21:30:38 UTC
Because of a few new dependencies, the patch doesn't apply anymore:

> CONFLICT (content): Merge conflict in C4/Installer/PerlDependencies.pm
Comment 26 David Cook 2018-04-10 00:02:53 UTC
Created attachment 73897 [details] [review]
Bug 18585 - Connect to RDF triplestore

This commit adds a 'triplestore' method to C4::Context. It takes
a model name as a parameter and returns a RDF::Trine::Model object
if triplestore.yaml is properly configured.

(At the moment, it just supports RDF::Trine::Store::SPARQL,
but it would be trivial to add support for RDF::Trine::Store::DBI too,
or any other RDF::Trine::Store::* backend.)

This code will provide a base from which people can use RDF::Trine
for querying and updating a RDF triplestore.