Bug 12620 - Proxy Add-on for Koha z39.50/SRU servers
Summary: Proxy Add-on for Koha z39.50/SRU servers
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: System Administration (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-07-22 01:32 UTC by Dan
Modified: 2024-04-24 10:37 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Allows to search in z39.50 servers using a proxy (6.13 KB, patch)
2024-04-24 10:37 UTC, Adolfo Rodríguez Taboada
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Dan 2014-07-22 01:32:24 UTC
Hi all,
 I am new to Koha and have set it up to use in a Victorian School. Great product and congrats on all the community work.

With Vic schools we all have to run through an upstream proxy and as such we don't have a 'direct' internet connection.

We wish to use the z39.50 module to download information from SCIS however whenever we try to sync it lists network errors(basically implying that there is no internet connection).

After a bit of reading and searching the only way to get around this appears to be to use YAZProxy - however I was wondering if there is any way in upcoming releases to have a proxy configuration section.
Doing this would make it a more feasible and easier to implement solution for schools in Australia (if not around the world).

Any input would be great.

Thanks
Dan
Comment 1 Sistemas PSIUBA 2014-10-16 21:09:36 UTC
+1 

I would use the "vote" feature but right now, it's not working.
Comment 2 Galen Charlton 2015-10-28 16:10:46 UTC
Depending on the nature of the proxy involved, it may not be necessary to use YAZProxy. In particular, the YAZ toolkit supports a special connection syntax to direct Z39.50 and SRU traffic through a proxy.

For example, suppose you have a Squid proxy running on localhost and listening on port 210, and further suppose that the proxy is configured to allow connecting to the target Z39.50 server(s).  In that case, with yaz-client you can do something like this:

yaz-client connect:localhost:3128,tcp:lx2.loc.gov:210/LCDB

and that will get you a proxied Z39.50 connection to the Library of Congress.

As another example, the following would open a proxied connection to the GBV (Gemeinsame Bibliotheksverbund) SRU server:

yaz-client connect:localhost:3128,http://sru.gbv.de:80/gvk

Translating that to Koha, a system preference or a new column in the z3950servers table could be added to specify a proxy to use.
Comment 3 Dan 2018-10-12 04:25:44 UTC
Hi all,
Just wanted to see if a proxy function within Koha is being investigated?

Thank you for your ongoing efforts.
Comment 4 Katrin Fischer 2018-10-20 11:35:19 UTC
Hi Dan, new features in Koha are usually added when someone sponsors the feature. Most developers work for support companies developing features as part of their work. That said, you could also try and bring up your question on the mailing list to find others interested in the feature or maybe even someone who has found a workaround or is working on a solution.
Comment 5 Katrin Fischer 2021-06-03 15:09:42 UTC
Reading my own comment from 2018 and just having run into this issue - is anyone aware of a workaround for this problem?
Comment 6 David Cook 2021-06-04 01:29:45 UTC
This is an interesting one, and Dan is in Australia as well, so I'm extra interested.

I'd need to know more about the network ecosystem at the school, but it sounds like all outbound network traffic is blocked, unless it uses a HTTP forward proxy. Of course, that means that they'll be restricted to only using HTTP (or maybe just ports 80 and 443 depending on the type of proxy). 

Since Z39.50 tends to use ports 210 or 7090, that's not going to be an option. 

SRU could be an option, since it uses HTTP, but the SCIS website seems to suggest it only has Z39.50 for programmatic access (https://www.scisdata.com/products/scis-data/). 

Looking at https://metacpan.org/pod/ZOOM and https://software.indexdata.com/yaz/doc/zoom.html, there is a proxy option, so if it's not a HTTP forward proxy and is just a TCP proxy... then that could be investigated as an option. 

Happy to chat with Dan in any case.
Comment 7 David Cook 2021-06-04 02:15:15 UTC
It looks like the "proxy" connect option expects the target to be a Z39.50 proxy.

However... maybe you could tunnel Z39.50 through HTTP...depending on the proxy...

https://en.wikipedia.org/wiki/HTTP_tunnel
https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods/CONNECT
https://datatracker.ietf.org/doc/html/rfc7231#section-4.3.6

It looks like the YAZ toolkit added something for that in 5.10.0 in 2015: https://github.com/indexdata/yaz/blob/master/NEWS#L457

https://github.com/indexdata/yaz/commit/9021ba4aa6fcdbdd177073d44edd03a86d70198f

I'm finding the YAZ documentation to be quite poor... so I'm going to reach out to Indexdata and see what they have to say on the topic.

This might actually be possible...
Comment 8 David Cook 2021-06-04 03:08:28 UTC
(In reply to Galen Charlton from comment #2)
> Depending on the nature of the proxy involved, it may not be necessary to
> use YAZProxy. In particular, the YAZ toolkit supports a special connection
> syntax to direct Z39.50 and SRU traffic through a proxy.
> 
> For example, suppose you have a Squid proxy running on localhost and
> listening on port 210, and further suppose that the proxy is configured to
> allow connecting to the target Z39.50 server(s).  In that case, with
> yaz-client you can do something like this:
> 
> yaz-client connect:localhost:3128,tcp:lx2.loc.gov:210/LCDB
> 
> and that will get you a proxied Z39.50 connection to the Library of Congress.

It took a bit of digging but I found some information about that syntax that Galen mentioned:

https://github.com/indexdata/yaz/blob/master/NEWS#L2365

So in that case you'd be doing "yaz-client connect:EDUPROXY:PROXYPORT,tcp:lx2.loc.gov:210/LCDB".

Now ZOOM doesn't explicitly mention support for this syntax (https://metacpan.org/pod/distribution/Net-Z3950-ZOOM/lib/ZOOM.pod). Of course, neither does yaz-client (https://linux.die.net/man/1/yaz-client). As I said, the documentation leaves something to be desired.

I'm waiting to hear back from Adam Dickmeiss about that. 

But looking at C4::Breeding::_create_connection... it looks like Koha would need a little patch to change how we pass in the connection string.

Instead of '$obj->connect( $server->{host}, $server->{port} )', we'd want to use something like '$obj->connect($host)' where $host is constructed based on whether or not there is a proxy. (See https://metacpan.org/pod/distribution/Net-Z3950-ZOOM/lib/ZOOM.pod#new())

I could keep digging through source code but I think it'll just be easier to wait for Adam's response.
Comment 9 David Cook 2021-06-04 03:10:50 UTC
Note though that this proxying of Z39.50 via a HTTP proxy will only work if your HTTP proxy allows connections to Z39.50 ports (like 210, 7090) and doesn't restrict them to 80 and 443.

But still... this has been very illuminating.
Comment 10 David Cook 2021-06-06 23:33:21 UTC
I've heard back from Adam Dickmeiss at Indexdata, and he said "you should be able to use [the connect syntax] for most client software of YAZ".

So that's interesting!

I'm tempted to do some testing, and build this functionality into Koha.
Comment 11 David Cook 2021-06-08 06:18:09 UTC
Adam inspired me to test this, and I got a proof-of-concept working in Docker using a Squid proxy. 

Here are the steps if someone wants to reproduce my efforts:

Windows:
1.	docker pull sameersbn/squid:3.5.27-2
2.	docker run --name squid --publish 3128:3128 sameersbn/squid:3.5.27-2
3.	docker cp squid:/etc/squid/squid.conf ./squid.conf.default
4.	cp ./squid.conf.default ./squid.conf
5.	sed -i 's/http_access deny all/http_access allow all/' ./squid.conf
6.	sed -i 's/http_access deny CONNECT/http_access allow CONNECT/' ./squid.conf
7.	docker stop squid
8.	docker rm squid
9.	docker run --name squid --publish 3128:3128 --volume "/$(PWD)/squid.conf:/etc/squid/squid.conf" --net koha_kohanet sameersbn/squid:3.5.27-2

Koha container:
1.	http_proxy=squid:3128 curl -v https://www.google.com
2a.	yaz-client connect:squid:3128,tcp:lx2.loc.gov:210
2b.	base LCDB
2c.	find test
2d.	show 1

3a.	Use ZOOM in a Perl script to connect to Squid proxy
3b.	This works (shows 10000 results connecting to LOC LCDB database and searching for as title of ‘test’)
3c.     Script included below*

*
#!/usr/bin/perl
use ZOOM;
my $conn = new ZOOM::Connection('connect:squid:3128,tcp:lx2.loc.gov:210');
$conn->option( databaseName => 'LCDB' );
my $rs = $conn->search_pqf('@attr 1=4 test');
my $n = $rs->size();
warn $n;
Comment 12 David Cook 2021-06-08 06:48:42 UTC
Actually for readability we could change 'connect:squid:3128,tcp:lx2.loc.gov:210' to 'connect:squid:3128,lx2.loc.gov:210'.

To get this working for Koha...

We would need to change our syntax in C4::Breeding::_create_connection to use '$obj->connect($host)' instead of '$obj->connect( $server->{host}, $server->{port} )'.

We would also need some way to specify the HTTP Proxy. 

Option 1:
Add a http_proxy field (or http_proxy_hostname and http_proxy_port fields) to the z3950servers table. 

Option 2:
Specify a http_proxy in the koha-conf.xml. 

This might be a better option, since it's more centralized. Plus, there might be other parts of Koha that would benefit from being able to go through a HTTP proxy. (For example OverDrive integration?)
Comment 13 Adolfo Rodríguez Taboada 2024-04-24 10:37:02 UTC
Created attachment 165452 [details] [review]
Allows to search in z39.50 servers using a proxy

    In some environments it is necessary to use a proxy to search in z39.50 servers wether it's because it doesn't have access to Internet or that the z39.50 ports are blocked.
    This commit allows to configure a proxy in a new system preference HTTPS_Proxy.
    Sometimes some of the z39.50 servers don't work with the proxy, so we have added a column at server level so you can specify which ones won't use the proxy.
    To use the proxy the preference must be configured and the "Do not use proxy" checkbox in the z39.50 must be unchecked.
    
    Test plan
    1 Check searching agains z39.50 No results.
    2 Apply patch, restart services
    3 Configure system preference HTTPS_Proxy
    4 Check searching agains z39.50 and see that you get results.