Bug 30332

Summary: Outbound SRU does not work via https
Product: Koha Reporter: Thomas Klausner <domm>
Component: Z39.50 / SRU / OpenSearch ServersAssignee: Bugs List <koha-bugs>
Status: CLOSED WORKSFORME QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: dcook, domm, florian, m.de.rooy, nick
Version: unspecified   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: a simple proxy to talk with https SRUs

Description Thomas Klausner 2022-03-22 13:00:26 UTC
I tried to add an SRU service for https://www.obvsg.at/en/catalogues/access-via-sru. They only provide access via https. So I set it up like this:

Server name: OBVSGT
Hostname: https://services.obvsg.at
Port: 443
Database: sru/OBV-LIT
Additional SRU Options: operation=searchRetrieve, sru=get, sru_version=1.1
SRU search field mappings: title=alma.title,isbn=alma.isbn,srchany=alma.all_for_ui,author=alma.creator,issn=alma.issn,subject=alma.subjects,dewey=alma.dewey_decimal_class_number

This produces a valid URL when calling _create_connetion in C4::Breeding.

But the SRU API always returned errors: 235 Database does not exist.

So I tried to figure out what Koha is actually sending, but failed, because this seems to happen in some C code via ZOOM.

But we have a lot of working SRU setups via plain http. So I hacked up a quick proxy, launched that on a non-https port, and now it works.

So I assume that Zoom has a problem with https, which is very annoying, because more and more services will switch to https only in the future (or already have switched)

Any ideas what to do here?

I will attach my proxy script in a minute, so you can see for yourself...
Comment 1 Thomas Klausner 2022-03-22 13:02:59 UTC
Created attachment 132018 [details]
a simple proxy to talk with https SRUs

To start the proxy, run:

plackup -p 7007 mitmkoha.pl

Then change the SRU config to use
Hostname: localhost
Port: 7007
Database: /
Comment 2 Thomas Klausner 2022-03-22 13:05:31 UTC
It would be trivial to implement this proxy directly in Koha so we don't need another service, but as this is an ugly workaround I hope there are better solutions...
Comment 3 Marcel de Rooy 2022-03-22 13:09:21 UTC
(In reply to Thomas Klausner from comment #2)
> It would be trivial to implement this proxy directly in Koha so we don't
> need another service, but as this is an ugly workaround I hope there are
> better solutions...

Not sure what your proxy script does. But if you would use nginx as a reverse proxy between internet and Koha, I would not consider that an ugly solution. The SSL with its certificates etc is handled in the reverse proxy. Koha does not have to deal with that anymore. (We run Koha in a container at the same time.)
Comment 4 Thomas Klausner 2022-04-05 09:15:22 UTC
(In reply to Marcel de Rooy from comment #3)

> Not sure what your proxy script does. But if you would use nginx as a
> reverse proxy between internet and Koha, I would not consider that an ugly
> solution. The SSL with its certificates etc is handled in the reverse proxy.
> Koha does not have to deal with that anymore. (We run Koha in a container at
> the same time.)

Hm, yes, using an external server as a proxy would also be a good workaround. But (at least on one installation we manage) we don't have nginx (only Apache, which can of course also proxy), but the Apache config is managed by the IT department, which is far, far away and does not care about the library a lot. So changes there are rather hard to get..

Anyway, I've since improved the code a bit and put it into a plugin:

https://github.com/HKS3/HKS3-SRU-HTTPS-Proxy

This allows the librarians to add/edit/remove new HTTPS SRU Servers without any action either our or the IT departments part.

But this is still a hack, a proper solution would be to add HTTPS support to SRU, but I think this is not possible (or only with some work on the underlying ZOOM C code).
Comment 5 Marcel de Rooy 2022-04-05 11:28:58 UTC
(In reply to Thomas Klausner from comment #4)
> Hm, yes, using an external server as a proxy would also be a good
> workaround. But (at least on one installation we manage) we don't have nginx
> (only Apache, which can of course also proxy), but the Apache config is
> managed by the IT department, which is far, far away and does not care about
> the library a lot. So changes there are rather hard to get..

Yes, this sounds awkward.

> But this is still a hack, a proper solution would be to add HTTPS support to
> SRU, but I think this is not possible (or only with some work on the
> underlying ZOOM C code).

I would not recommend doing so. You'd better concentrate on running Koha in a container and put a (standard) nginx container or so before it as reverse proxy handling https.
By the way, you should be able to handle the same thing in Apache too without touching Zoom C code. Catch the https request, proxy it to the local http port where Zebra listens, etc.
Comment 6 Thomas Klausner 2022-04-06 13:29:09 UTC
> I would not recommend doing so. You'd better concentrate on running Koha in
> a container and put a (standard) nginx container or so before it as reverse
> proxy handling https.
> By the way, you should be able to handle the same thing in Apache too
> without touching Zoom C code. Catch the https request, proxy it to the local
> http port where Zebra listens, etc.

While I prefer nginx, Koha comes with Apache, and adding another webserver seems a bit like overkill.

And the problem is not related to Zebra or local code, but that some SRU servers only provide https access. And Koha cannot talk to those servers. It's not *incoming* https that's the problem!

And finally, this solution would make it impossible for people with only access to the web interface to add SRU services (because the proxy config would have to be set in nginx/apache config files)

Again, the proper solution would be to fix Koha to be able to use https SRU services. Currently this fails horrible, not even reporting any errors back, just "no hits".
Comment 7 Nick Clemens 2022-04-06 14:00:46 UTC
Testing with the example config you give I am able to connect and retrieve results when using KTD - so I am not sure this is a Koha issue

On the other hand, testing from the bywater koha demo - I get no results - so I wonder if it is a difference in master, or a configuration issue
Comment 8 Thomas Klausner 2022-04-06 14:28:17 UTC
(In reply to Nick Clemens from comment #7)
> Testing with the example config you give I am able to connect and retrieve
> results when using KTD - so I am not sure this is a Koha issue
> 
> On the other hand, testing from the bywater koha demo - I get no results -
> so I wonder if it is a difference in master, or a configuration issue

We're using 21.05, so of course this problem might have been fixed in HEAD? Though I didn't see any obvious commits regarding Zoom or SRU...

I think Katrin Fischer reported that accessing DNB SRU (german national library) also does not work when using https (but does with plain http). Not sure how to add her to this bug...

Thanks!
Comment 9 Katrin Fischer 2022-04-10 22:31:29 UTC
Usually I catch up on reading the bug list regularly. As I am CC on a lot of bugs that doesn't work quite as well for me.

I think it would be nice to get to the source of this, because as noted a solution that requires no server side changes would be best. 

The difference between the local installation and demo makes me curious... Maybe a difference between packages and gitified?
Comment 10 David Cook 2022-04-11 01:34:17 UTC
I think there's some confusion in these comments, and it took me some time to unpack what is going on.

To clarify, the problem is when a Koha web user is doing a Z39.50/SRU search of a remote host using SRU with HTTPS, right?

I'm going to take a quick look since this is the kind of problem that itches my brain...
Comment 11 David Cook 2022-04-11 01:34:40 UTC
Btw, Thomas, which version of Koha are you using?
Comment 12 David Cook 2022-04-11 01:45:19 UTC
(In reply to Thomas Klausner from comment #0)
> I tried to add an SRU service for
> https://www.obvsg.at/en/catalogues/access-via-sru. They only provide access
> via https. So I set it up like this:
> 
> Server name: OBVSGT
> Hostname: https://services.obvsg.at
> Port: 443
> Database: sru/OBV-LIT
> Additional SRU Options: operation=searchRetrieve, sru=get, sru_version=1.1
> SRU search field mappings:
> title=alma.title,isbn=alma.isbn,srchany=alma.all_for_ui,author=alma.creator,
> issn=alma.issn,subject=alma.subjects,dewey=alma.dewey_decimal_class_number
> 

I tried this in a custom Koha 21.11 (Ubuntu 20.04 OS and installed with Debian packages) and I had no problem. 

Using libyaz5 version 5.31.1-1.indexdata.

Searching for a Title of "test", I got 786 pages.
Comment 13 Thomas Klausner 2022-04-11 06:02:20 UTC
Yes, we have problems with outbound SRU, thanks for fixing the subject

(In reply to David Cook from comment #12)

> I tried this in a custom Koha 21.11 (Ubuntu 20.04 OS and installed with
> Debian packages) and I had no problem. 
> 
> Using libyaz5 version 5.31.1-1.indexdata.
> 
> Searching for a Title of "test", I got 786 pages.

We're using Koha 21.05.04 on a Debian 10 (buster) via debian packages, and libyaz5 5.27.1-2

So I guess we should at least try to upgrade libyaz5, because I did not see any relevant Koha commits between 21.05 and 21.11.

And these version differences could also explain why it works on some test instances, but not on others...

Greetings,
domm
Comment 14 Thomas Klausner 2022-04-11 06:08:53 UTC
OK, I found something in the YAZ changelog that might be relevant:

https://software.indexdata.com/yaz/doc/NEWS

> --- 5.30.3 2020/06/10
> 
> Require getaddrinfo which has been available since 2003.
> 
> Provide server host for SSL by using gnutls_server_name_set(3).
> Thx to Guillaume Jactat.

So I'll try to get the sysadmin to update to a more recent libyaz5, and also get them to provide us with a more recent debian (11, stable; instead of 10, oldstable).

If it's OK for you, I'll keep this ticket open until I have confirmed that a libyaz5 update is the way to go!

Greetings,
domm
Comment 15 David Cook 2022-04-11 07:13:16 UTC
(In reply to Thomas Klausner from comment #14)
> OK, I found something in the YAZ changelog that might be relevant:
> 
> https://software.indexdata.com/yaz/doc/NEWS
> 
> > --- 5.30.3 2020/06/10
> > 
> > Require getaddrinfo which has been available since 2003.
> > 
> > Provide server host for SSL by using gnutls_server_name_set(3).
> > Thx to Guillaume Jactat.
> 
> So I'll try to get the sysadmin to update to a more recent libyaz5, and also
> get them to provide us with a more recent debian (11, stable; instead of 10,
> oldstable).
> 
> If it's OK for you, I'll keep this ticket open until I have confirmed that a
> libyaz5 update is the way to go!
> 
> Greetings,
> domm

Yeah that sounds reasonable to me. Plus, older versions of Debian are still listed as recommended for Koha on https://wiki.koha-community.org/wiki/System_requirements_and_recommendations so it would be good to determine that the YAZ library is the culprit here so we could note it down.
Comment 16 Florian Saling 2022-08-08 16:54:43 UTC
(In reply to David Cook from comment #15)
> (In reply to Thomas Klausner from comment #14)
> > OK, I found something in the YAZ changelog that might be relevant:
> > 
> > https://software.indexdata.com/yaz/doc/NEWS
> > 
> > > --- 5.30.3 2020/06/10
> > > 
> > > Require getaddrinfo which has been available since 2003.
> > > 
> > > Provide server host for SSL by using gnutls_server_name_set(3).
> > > Thx to Guillaume Jactat.
> > 
> > So I'll try to get the sysadmin to update to a more recent libyaz5, and also
> > get them to provide us with a more recent debian (11, stable; instead of 10,
> > oldstable).
> > 
> > If it's OK for you, I'll keep this ticket open until I have confirmed that a
> > libyaz5 update is the way to go!
> > 
> > Greetings,
> > domm
> 
> Yeah that sounds reasonable to me. Plus, older versions of Debian are still
> listed as recommended for Koha on
> https://wiki.koha-community.org/wiki/System_requirements_and_recommendations
> so it would be good to determine that the YAZ library is the culprit here so
> we could note it down.

The upgrade to Debian 11 from Debian 9 solved our problems with SRU services requiring HTTPS. Debian 9 had libyaz5 5.27.1 installed and Debian 11 shipped with 5.30.3. So I think it is safe to assume that the YAZ library was the culprit!
Comment 17 David Cook 2022-08-08 23:51:46 UTC
(In reply to Florian Saling from comment #16)
> The upgrade to Debian 11 from Debian 9 solved our problems with SRU services
> requiring HTTPS. Debian 9 had libyaz5 5.27.1 installed and Debian 11 shipped
> with 5.30.3. So I think it is safe to assume that the YAZ library was the
> culprit!

Good to know! I'll make a note on https://wiki.koha-community.org/wiki/System_requirements_and_recommendations
Comment 18 David Cook 2022-08-08 23:53:58 UTC
Are folk happy to close this one now? 

See https://wiki.koha-community.org/wiki/System_requirements_and_recommendations#Zebra
Comment 19 Katrin Fischer 2022-11-17 00:26:04 UTC
(In reply to David Cook from comment #18)
> Are folk happy to close this one now? 
> 
> See
> https://wiki.koha-community.org/wiki/
> System_requirements_and_recommendations#Zebra

I'd say so :)