Bug 21210

Summary: Elasticsearch based SRU server
Product: Koha Reporter: Joonas Kylmälä <joonas.kylmala>
Component: Z39.50 / SRU / OpenSearch ServersAssignee: Ere Maijala <ere.maijala>
Status: RESOLVED DUPLICATE QA Contact: Testopia <testopia>
Severity: new feature    
Priority: P5 - low CC: ere.maijala, johanna.raisa, katrin.fischer, m.de.rooy, magnus
Version: unspecifiedKeywords: dependency
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=13937
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 13937    
Bug Blocks:    
Attachments: Bug 21210: Implement Elasticsearch based SRU server

Description Joonas Kylmälä 2018-08-13 08:18:51 UTC
There is no SRU server yet that works with Koha's Elasticsearch index. Without a Elasticsearch based SRU server one would have to run Zebra alongside Elasticsearch and that has it downsides like added complexity and non-realtime updates to the Zebra index (because the index will be only updated to Elasticsearch on biblio updates). 

There has been some work on the Elasticsearch SRU server: https://github.com/NatLibFi/Koha/blob/ks-rumble/misc/sru_server.pl. Maybe that could be adapted to the current version of Koha?
Comment 1 Joonas Kylmälä 2018-08-14 13:49:04 UTC
Created attachment 77787 [details] [review]
Bug 21210: Implement Elasticsearch based SRU server

I combined the previous work by Ere together and based it on the latest master.

I would like to know if you have any opinions about storing the "cql_mapping" in /etc/sru_server/catmandu.yml. Should we just treat this file in a way that we know everybody must tweak it to their liking and just do a configuration that has most basic fields like author and title?

General comments about the implementation are also welcome.
Comment 2 Ere Maijala 2018-09-15 21:30:03 UTC
Thanks for doing this, Joonas. I'll be happy to do any changes that may be needed to make everything work. I suppose we may need to put the default config file somewhere else than /etc since that caused trouble with other ES files too.

In a more perfect world I think we should have a search backend agnostic SRU server. That's a fair bit more work to implement, though.
Comment 3 Johanna Räisä 2018-11-20 09:29:41 UTC
This is missing few Perl dependecies. Please include Dancer and Dancer::Plugin::Catmandu::SRU. Otherwise first impression is good.
Comment 4 Katrin Fischer 2018-11-20 10:49:32 UTC
(In reply to Johanna Räisä from comment #3)
> This is missing few Perl dependecies. Please include Dancer and
> Dancer::Plugin::Catmandu::SRU. Otherwise first impression is good.

As Dancer is a whole framework I think we should probably discuss in a dev meeting. We've had similar discussions about including other frameworks in the past (I remember Moose and Moo)
Comment 5 Magnus Enger 2018-11-20 10:58:36 UTC
If there are site-specific config files those should probably go in /etc/koha/sites/<instance>/, at least for Debian installs?

(In reply to Katrin Fischer from comment #4)
> (In reply to Johanna Räisä from comment #3)
> > This is missing few Perl dependecies. Please include Dancer and
> > Dancer::Plugin::Catmandu::SRU. Otherwise first impression is good.
> 
> As Dancer is a whole framework I think we should probably discuss in a dev
> meeting. We've had similar discussions about including other frameworks in
> the past (I remember Moose and Moo)

Dancer is a web-framework, more comparable to Mojolicious, I think. And we are already using Mojo for the REST API. Would rewriting the SRU server to use Mojo be doable? 

An SRU server for Elasticsearch would definitely be awesome, so thanks for working on this!
Comment 6 Ere Maijala 2018-11-20 11:01:31 UTC
From what I gathered from the discussion in Kohacon, this would be only a stop-gap solution and we should rather aim for something more search backend agnostic and support both SRU and Z39.50. I may have some code that might be useful, but I need a bit of time to work on it.
Comment 7 Marcel de Rooy 2018-11-20 13:18:10 UTC
Supporting Z39.50 and SRU on Elastic side is something we really need imo to abandon Zebra completely.
Comment 8 Ere Maijala 2018-11-20 13:49:18 UTC
Note also that AFAIK we can't use Plack for the server since Z39.50 is not HTTP-based. I'm leaning towards something like Net::Z3950::SimpleServer where the daemon would work in a similar fashion to the SIP server in Koha.
Comment 9 Magnus Enger 2018-11-20 13:50:52 UTC
(In reply to Ere Maijala from comment #8)
> Note also that AFAIK we can't use Plack for the server since Z39.50 is not
> HTTP-based. I'm leaning towards something like Net::Z3950::SimpleServer
> where the daemon would work in a similar fashion to the SIP server in Koha.

Do we need to have a Z39.50 server? At least in Norway that would not be a requirement, I think.
Comment 10 Marcel de Rooy 2018-11-20 13:54:11 UTC
(In reply to Magnus Enger from comment #9)
> (In reply to Ere Maijala from comment #8)
> > Note also that AFAIK we can't use Plack for the server since Z39.50 is not
> > HTTP-based. I'm leaning towards something like Net::Z3950::SimpleServer
> > where the daemon would work in a similar fashion to the SIP server in Koha.
> 
> Do we need to have a Z39.50 server? At least in Norway that would not be a
> requirement, I think.

Yeah we still need it. Think about all copy cataloging that still goes thru Z39.50 and not SRU.
Comment 11 Ere Maijala 2018-11-21 12:26:07 UTC
Since there's already work done on a SimpleServer in bug 13937, it would probably make sense to use it as the basis.
Comment 12 Ere Maijala 2018-11-23 07:52:07 UTC
I'm marking this duplicate of bug 13937 since the implementation there will work with any search engine and doesn't need Dancer. I still need to make sure we have all the index definitions that were available here, but the current version in bug 13937 is already functional.

*** This bug has been marked as a duplicate of bug 13937 ***