Bug 25965 - Create SIP2 client daemon with HTTPS API endpoint
Summary: Create SIP2 client daemon with HTTPS API endpoint
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: REST API (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low new feature (vote)
Assignee: Bugs List
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-07-09 12:45 UTC by Lari Strand
Modified: 2023-12-14 23:31 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Patch for adding SIP2 over HTTPS support (12.33 KB, patch)
2020-09-17 13:47 UTC, Lari Strand
Details | Diff | Splinter Review
Screenshot of Postman communicating with sip server via /sipmessages endpoint using xml (131.21 KB, image/png)
2020-09-17 14:00 UTC, Lari Strand
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Lari Strand 2020-07-09 12:45:10 UTC
I am trying to pass messages to a sip server using a new endpoint I created in swagger/openApi. This endpoint uses the guidelines described here:

http://biblstandard.dk/rfid/dk/rfid_sip2_over_https.htm

The idea is to get rid of the tunneling/VPN implementations that we use, and use HTTPS instead to pass sip messages back and forth inside an XML. The Apache server handles the HTTPS stuff so I don't have had to worry about that so far when communicating with the REST api, hence the SipOHttp naming in my module.
At the moment I can send the XML POST messages formed like in the linked guideline to the endpoint (XML wrapped SIP2 message to .../api/v1/sipmessages/). The api forwards the body content of the request to my module Koha::REST::V1::SipOHttp. The module validates and parses the contents of the XML message, extracts the information about the sip server the message is intended to (using the "login:" parameter inside the XML) and the sip message itself (inside the <request></request> variable). Then the module passes the sip message on to the sip server using IO:Socket::INET.

My problem is that the sip server closes the connection after this first SC device login message has been passed and the response has been read back in my module:

(from sip2.log)
.....
[INFO] [undef]@10.0.3.217: OUTPUT MSG: '941' C4::SIP::Sip::write_msg /home/koha/Koha/C4/SIP/Sip.pm (187) 
[DEBUG] SIPDEVICE@10.0.3.217: raw_transport: uname/inst: 'SIPDEVICE/SIPDEVICE' C4::SIP::SIPServer::raw_transport /home/koha/Koha/C4/SIP/SIPServer.pm (174)
[DEBUG] SIPDEVICE@10.0.3.217: EOF returned on read C4::SIP::SIPServer::read_request /home/koha/Koha/C4/SIP/SIPServer.pm (349) 
[INFO] SIPDEVICE@10.0.3.217: raw_transport: shutting down C4::SIP::SIPServer::raw_transport /home/koha/Koha/C4/SIP/SIPServer.pm (178)

So I need a way to keep this socket connection open for other future messages. Without touching the sip server's code itself, the only solution I can come up with is a proxy server that handles and passes on the REST api's incoming messages to sockets and keeps each invidual socket connection open to specific instances of the sip servers (I suppose the sip server itself ends the connections when needed so you wouldn't have to worry about it in the sender's end).
Another solution would be to build a new api separate from the swagger/Openapi one. Like an all-in-one solution that deals with the requests and keeps tabs on the open socket connections.

Let me know if you have ideas regarding this.

Ps. I'm a trainee in Koha Suomi (Finland) so excuse me if the description of my problem is a bit vague and lacks some correct terms used. I'm happy to fill in any details I've left out.
Comment 1 Lari Strand 2020-07-09 13:52:45 UTC
Or is there a way to leave the sip server socket connection open for future messages, in a way that it still allows me to catch the sip server's response, and when another message is being sent in the same address, check if the connection already exists and use that instead of creating a new socket connection, directly in the module?
Comment 2 David Cook 2020-07-10 06:00:51 UTC
(In reply to Lari Strand from comment #0)
> I am trying to pass messages to a sip server using a new endpoint I created
> in swagger/openApi. This endpoint uses the guidelines described here:
> 
> http://biblstandard.dk/rfid/dk/rfid_sip2_over_https.htm
> 

I find it odd that they say "SIP2 messages are typically exchanged directly over tcp or via telnet, but this prevents SIP2 being used over an internet connection". The Internet is made up of predominately TCP/IP networks. Maybe they meant "the World Wide Web" in regards to HTTP(S) specifically? 

I don't understand why they'd have to use HTTPS. What are the target SIP2 client and SIP2 server technologies being used? Is this for Koha's SIP2 server? What's going to be acting as the SIP2 client?

> My problem is that the sip server closes the connection after this first SC
> device login message has been passed and the response has been read back in
> my module:
> 

> So I need a way to keep this socket connection open for other future
> messages.

Koha's SIP2 server will have a TCP socket in a LISTEN state. The SIP2 client will establish a TCP connection and when the SIP2 server accepts that connection, it creates a new socket which lasts for the duration of that connection (you could view this using system utilities like 'netstat' or 'ss', the latter being more modern and useful). If either the server or the client close their end of that socket, that socket is finished.

> Without touching the sip server's code itself, the only solution I
> can come up with is a proxy server that handles and passes on the REST api's
> incoming messages to sockets and keeps each invidual socket connection open
> to specific instances of the sip servers (I suppose the sip server itself
> ends the connections when needed so you wouldn't have to worry about it in
> the sender's end).
> Another solution would be to build a new api separate from the
> swagger/Openapi one. Like an all-in-one solution that deals with the
> requests and keeps tabs on the open socket connections.
> 

Does it have to be HTTPS?

You should be able to use Nginx as a TCP proxy with SSL termination. (I don't know about Apache's abilities as a TCP proxy. I haven't heard of it, but it could be a thing. I only have personal experience using Nginx as a TCP proxy for non-HTTP application protocols.)

Client:
https://metacpan.org/pod/IO::Socket::SSL
Proxy:
https://docs.nginx.com/nginx/admin-guide/security-controls/terminating-ssl-tcp/

Remember that HTTP is just an application protocol which uses TCP for its transport, and the TLS/SSL is done at the transport level. 

But... it sounds like there might be a bit of a sunken cost in terms of doing this SIP2 over HTTPS as per http://biblstandard.dk/rfid/dk/rfid_sip2_over_https.htm. 

If you have to go HTTPS, I'd say use either Koha or Apache to proxy the HTTPS to a backend service that has a HTTP(S) listener, uses sessions, and can create persistent connections to the SIP2 server.
Comment 3 David Cook 2020-07-10 06:06:54 UTC
Alternatively, I guess you could use a message queue like RabbitMQ.

The HTTPS client could add messages to 1 queue and listen for responses on a different queue. 

But you'd need some way to identify the HTTPS client for the queues. 

Overall, trying to send SIP2 over HTTPS sounds like a bit of a nightmare.
Comment 4 David Cook 2020-07-10 06:10:40 UTC
I'd be curious to hear why Koha Suomi (Finland) are opposed to stunnel as well.

I know many Koha libraries use stunnel, and a quick Google suggests other library software does too:

https://www.stunnel.org/
https://developers.exlibrisgroup.com/alma/integrations/selfcheck/
https://github.com/folio-org/edge-sip2
Comment 5 David Cook 2020-07-10 06:17:19 UTC
I have 1 more (non-HTTP) idea and that would actually be to integrate TLS/SSL support into the SIP2 server itself.

Koha uses Net::Server::PreFork for the SIP2 server, and that's a subclass of Net::Server, which looks like it has support for ssl/tcp sockets.

I reckon that would be more valuable than trying to send SIP2 over HTTPS.
Comment 6 Lari Strand 2020-09-17 13:47:58 UTC
Created attachment 110265 [details] [review]
Patch for adding SIP2 over HTTPS support
Comment 7 Lari Strand 2020-09-17 13:57:08 UTC
I managed to write a script for communicating with the SIP server through the REST api with XML messages.

Attached is a patch and a screenshot from Postman exchanging a SIP message.
Comment 8 Lari Strand 2020-09-17 14:00:15 UTC
Created attachment 110267 [details]
Screenshot of Postman communicating with sip server via /sipmessages endpoint using xml
Comment 9 David Cook 2020-09-17 23:10:04 UTC
If I understand correctly, Lari, there is a vendor that's is creating or has already created a SIP2 client that will use this HTTPS endpoint, right?

Is it possible for people to get more details about that?
Comment 10 David Cook 2020-09-17 23:18:23 UTC
I notice a few hard-coded references to koha-suomi.fi, so those would need to be removed and use replaceable placeholders.

You'll want to take a look at the coding guidelines too (https://wiki.koha-community.org/wiki/Coding_Guidelines), as I think you've used tabs here instead of 4 spaces (per tab). 

For the config, you might want to put it into koha-conf.xml as well (even if I do think that we overload that 1 configuration file with everything...)

Other than that, can you give us a test plan for how to get this to work?

Ideally, it would be great if it could work in koha-testing-docker (https://gitlab.com/koha-community/koha-testing-docker). 

Overall, it's looking good. Thanks for posting it!
Comment 11 Lari Strand 2020-09-18 06:55:45 UTC
(In reply to David Cook from comment #9)
> If I understand correctly, Lari, there is a vendor that's is creating or has
> already created a SIP2 client that will use this HTTPS endpoint, right?
> 
> Is it possible for people to get more details about that?

Well there's a finnish self check machine vendor here who agreed to test the new solution with us. It's been tested on a test server successfully. I can't get my hands on their code and they won't share it, but in the discussions we've had they've mentioned that this is the way they communicate with some other systems as well (originally the POST request xml needed to be in escape form, and they requested that the xml could be sent unescaped for a more unified solution, without a need for a Koha specific solution).
Comment 12 Lari Strand 2020-09-18 07:06:14 UTC
(In reply to David Cook from comment #10)
> I notice a few hard-coded references to koha-suomi.fi, so those would need
> to be removed and use replaceable placeholders.

That's true, the contents inside the xml elements are being validated and need to match the xml schema exactly. I'll work on that.

> 
> You'll want to take a look at the coding guidelines too
> (https://wiki.koha-community.org/wiki/Coding_Guidelines), as I think you've
> used tabs here instead of 4 spaces (per tab). 

I used Eclipse's formatting tool to format the code. I wasn't aware you can't use tabs. I'll remove the tabs in an updated version.

> 
> For the config, you might want to put it into koha-conf.xml as well (even if
> I do think that we overload that 1 configuration file with everything...)
> 
> Other than that, can you give us a test plan for how to get this to work?

I'll write a short description on how to set this up.

> 
> Ideally, it would be great if it could work in koha-testing-docker
> (https://gitlab.com/koha-community/koha-testing-docker). 
> 
> Overall, it's looking good. Thanks for posting it!

I'll check on the koha-testing-docker stuff, can't make any promises :)
Comment 13 Lari Strand 2020-09-18 07:55:52 UTC
Here a quick how-to:

For this module to work you obviously need to apply the patch. Then there's a configuration file called sip2ohttp-conf.xml attached that needs to be put into the same configuration file folder as the environment variable KOHA_CONF points to (where koha-conf.xml file is located).

The xml validation requires an xml schema file (sipschema.xml). The schema file should be located in the koha-tmpl folder (the parent directory of koha-tmpl folder gets read like this: C4::Context->config('intranetdir');) When the REST endpoint receives an XML message, it will be validated against this schema file before further processing.

In the configuration file sip2ohttp-conf.xml there's a short description on how to use it. You need to add the devices that you want to use sipohttps with here. The module SIPoHTTP.pm will search for device parameters from here when the REST endpoint /sipmessages receives a POST request with an XML in the request body.

example config parameters for a device login with an ID "SIPDEVICE" (also configured in SIPdevices.xml by the same ID):

	  	<SIPDEVICENAME>
			<host>127.0.0.1</host>
			<port>6009</port>
		</SIPDEVICENAME>

Host and port are the sip server's address you want the device to talk with. Obviously you need a sip server running in that address.

As mentioned, the device should also be configured to be able to use the sip servers in the SIPconfig.xml file. The device login id's configured in the sip2ohttp-conf.xml file need to match the login id's defined in SIPconfig.xml.

That's about it. Now you can communicate with the sip server/servers via the REST endpoint by passing it an XML file like in the Postman attachment I included.
Comment 14 Lari Strand 2020-09-18 07:59:45 UTC
typo: example config parameters for a device login with an ID "SIPDEVICENAME" ...
Comment 15 Lari Strand 2020-09-18 08:49:50 UTC
(In reply to David Cook from comment #10)
> I notice a few hard-coded references to koha-suomi.fi, so those would need
> to be removed and use replaceable placeholders.

You mean just like xsi:schemaLocation="<INSERTYOUROWNXMLSCHEMALOCATIONHERE>"?
The values could be anything, once I figure out how to only validate the xml structure, not the insides of the elements as well.

> For the config, you might want to put it into koha-conf.xml as well (even if
> I do think that we overload that 1 configuration file with everything...)

You mean that I should ditch the whole sip2ohttp-conf.xml file altogether and add like an example configuration in the koha-conf.xml file and make the module read the configs from that file instead? Wouldn't it be more easily accessible and simpler, if the sipohttps config would be a separate file? If we added dozens of device configs there the file would become even more bloated. Imagine putting the stuff in SIPconfig.xml into koha-conf.xml, almost the same thing. I would like to keep it separate.
Comment 16 Kyle M Hall 2020-09-21 13:21:20 UTC
This is a very interesting concept, but completely non-standard and I would not expect to see any adoption of it or even much use outside of you specific needs. NCIP has pretty much supplanted SIP for this use case ( NISO is still trying to figure out what to do with SIP & NCIP now that it controls both specifications ).

I think this would be much better suited as an API plugin.
Comment 17 David Cook 2020-09-21 23:52:44 UTC
(In reply to Kyle M Hall from comment #16)
> This is a very interesting concept, but completely non-standard and I would
> not expect to see any adoption of it or even much use outside of you
> specific needs. NCIP has pretty much supplanted SIP for this use case ( NISO
> is still trying to figure out what to do with SIP & NCIP now that it
> controls both specifications ).
> 
> I think this would be much better suited as an API plugin.

This sounds reasonable to me as well. 

I think that Lari has already implemented this locally (in testing) at Koha-Suomi in Finland, but I think that they'd probably benefit from doing it as an API plugin too. 

I suggested sharing the work on Bugzilla anyway though in case anyone else finds the work interesting. It also gives a chance to get some community feedback. 

Lari, is there a reason the vendor isn't using NCIP? (I admit that I've never used it so didn't even think of it. Cheers, Kyle!)
Comment 18 Lari Strand 2020-09-23 06:23:06 UTC
(In reply to David Cook from comment #17)

> > I think this would be much better suited as an API plugin.
> 
> This sounds reasonable to me as well. 
> 

I've been studying the plugins and indeed it would be fairly painless to build it into a plugin. I'm not sure if I have the time to do a plugin version, since my traineeship period is coming to it's end soon. Maybe on my free time.

> I think that Lari has already implemented this locally (in testing) at
> Koha-Suomi in Finland, but I think that they'd probably benefit from doing
> it as an API plugin too. 
> 
> I suggested sharing the work on Bugzilla anyway though in case anyone else
> finds the work interesting. It also gives a chance to get some community
> feedback. 
> 
> Lari, is there a reason the vendor isn't using NCIP? (I admit that I've
> never used it so didn't even think of it. Cheers, Kyle!)

I've never heard of NCIP as well. I was tasked to come up with a solution like this, based on the Axiell guideline I've posted in the first comment by Koha Finland so I did. So from what I can figure, there are already numerous sip device manufacturers that use this method of communication, at least here in the Nordics.
The point was not to touch the sip server or the way we currently talk with sip devices, but to introduce and add a new simpler way to pass messages to sip servers with no need for intermediate tunneling setups or VPN solutions since we use https (I have no idea how tunneling works, but I know it and VPN both need extra setup and management and VPN is not free?).
Here every passed message to the sip server/s would also be authenticated on a message by message basis. One need for this kind of a solution would be in the library buses with spotty and unreliable internet connections.

I'm not familiar with NCIP or how the security of the communications has been resolved with NCIP. Couldn't find anything with a quick google. If it makes tunneling/VPN stuff obsolete, then that's great. Just what we were after.

3M SIP2 protocol is now NISO property? Cool. Maybe it will now get updated to reflect current needs.
Comment 19 Lari Strand 2020-09-23 06:35:48 UTC
Also what comes to the plugin development, we don't have support for api endpoint injection yet. We are way behind in merging community stuff to our local development. My Koha version in my developer container says 17.06.00.001, but there's a lot of sftuff added since. But plugin support is outdated for sure. So I'm struggling on how to set up a community dev environment where I could continue work with the plugin version.
Comment 20 Kyle M Hall 2020-09-23 11:33:49 UTC
> I'm not familiar with NCIP or how the security of the communications has
> been resolved with NCIP. Couldn't find anything with a quick google. If it
> makes tunneling/VPN stuff obsolete, then that's great. Just what we were
> after.

NCIP uses HTTP for communication, so it supports SSL/TLS. It's XML based: https://www.niso.org/standards-committees/ncip
 
> 3M SIP2 protocol is now NISO property? Cool. Maybe it will now get updated
> to reflect current needs.

Yep, https://www.niso.org/standards-committees/sip
I've been waiting years for SIP3. At this point I'm not holding my breath ;)

Our NCIP implementation can be found here: https://github.com/bywatersolutions/ncip-server
It does not support the complete NCIP message set, but it's quite easy to add messages as they are needed.
Comment 21 Lari Strand 2022-10-28 12:28:04 UTC
Koha-Suomi SIPoHTTP (SIP2 over HTTPS) communication plugin can be found at:

https://github.com/KohaSuomi/koha-plugin-SIPoHTTP

It's in full production use. It uses the same old SIP2 communication protocol and has required software updates for machines that we wanted to communicate thourgh it.
We still use a couple of tunneled connections to SIP-servers but the plugin works nicely next to those pre-existing SIP2-server configurations. The plugin creates a new endpoint which accepts XML messages with their embedded SIP commands and passes them to Koha's SIP servers. The plugin relays the replies via a response XML in the response body. The documentation on how to configure/use it is in Finnish though. Take a look if you're interested.
Comment 22 David Cook 2022-10-31 00:39:21 UTC
(In reply to Lari Strand from comment #21)
> Koha-Suomi SIPoHTTP (SIP2 over HTTPS) communication plugin can be found at:
> 
> https://github.com/KohaSuomi/koha-plugin-SIPoHTTP

Cool!

I'll certainly keep it in mind for the future.
Comment 23 Katrin Fischer 2023-12-09 16:41:51 UTC
Hi Lari, does you plugin cover the full functionality or are there any changes needed in Koha still? 

We could add it to https://wiki.koha-community.org/wiki/Koha_plugins and close the bug maybe.
Comment 24 Lari Strand 2023-12-11 07:40:43 UTC
(In reply to Katrin Fischer from comment #23)
> Hi Lari, does you plugin cover the full functionality or are there any
> changes needed in Koha still? 
> 
> We could add it to https://wiki.koha-community.org/wiki/Koha_plugins and
> close the bug maybe.

It covers the full functionality of SIP2 and acts as a wrapper for talking to the Koha SIP2 server. Some changes to the SIP server configuration file is needed, nothing else. So yes it works as a "standalone" plugin. I guess we could close the ticket.
Comment 25 Katrin Fischer 2023-12-11 09:14:08 UTC
(In reply to Lari Strand from comment #24)
> (In reply to Katrin Fischer from comment #23)
> > Hi Lari, does you plugin cover the full functionality or are there any
> > changes needed in Koha still? 
> > 
> > We could add it to https://wiki.koha-community.org/wiki/Koha_plugins and
> > close the bug maybe.
> 
> It covers the full functionality of SIP2 and acts as a wrapper for talking
> to the Koha SIP2 server. Some changes to the SIP server configuration file
> is needed, nothing else. So yes it works as a "standalone" plugin. I guess
> we could close the ticket.

Please do :)
Comment 26 Marcel de Rooy 2023-12-11 09:16:21 UTC
I have the idea that the reason behind this report is still valid. People want to protect their SIP2 communication and this is still not available in standard Koha. But it might deserve a report on its own?
Comment 27 Kyle M Hall 2023-12-11 13:59:22 UTC
(In reply to Marcel de Rooy from comment #26)
> I have the idea that the reason behind this report is still valid. People
> want to protect their SIP2 communication and this is still not available in
> standard Koha. But it might deserve a report on its own?

Unfortunately this won't protect or work with standard SIP2. SIP2 over http is not a commonly supported feature as far as I am aware. The standard for protecting regular SIP2 traffic would be stunnel. Many if not most SIP2 devices have stunnel support built in. As far as I can tell, there is no way to bake stunnel into our SIP server.
Comment 28 David Cook 2023-12-11 22:16:13 UTC
(In reply to Kyle M Hall from comment #27)
> (In reply to Marcel de Rooy from comment #26)
> > I have the idea that the reason behind this report is still valid. People
> > want to protect their SIP2 communication and this is still not available in
> > standard Koha. But it might deserve a report on its own?
> 
> Unfortunately this won't protect or work with standard SIP2. SIP2 over http
> is not a commonly supported feature as far as I am aware. The standard for
> protecting regular SIP2 traffic would be stunnel. Many if not most SIP2
> devices have stunnel support built in. As far as I can tell, there is no way
> to bake stunnel into our SIP server.

Yeah this is apples and oranges. 

Bug 25965 is about adding support for HTTPS which wouldn't be helpful for 99.99% of libraries. 

We did have bug 25972 for adding SSL/TLS support to the SIP server, but ended up closing it, since stunnel is an already working industry standard. That said, if someone did want to build SSL/TLS support into our SIP server, I don't think it would be very hard at all.
Comment 29 Lari Strand 2023-12-12 10:31:16 UTC
Lyngsoe systems and Mikroväylä added support for their self-check machines for our SIPoHTTPS plugin system here in Finland. We have hundreds of machines here that talk with our Koha SIP servers, including door systems, smart book cabinets, etc. The need for this came because of the near impossibility to set up and manage stunnels for all devices without problems. Now we can maintain and reconfigure our server side freely without affecting the self check machines or need for reconfiguration for the stunnels. The self-check machines also work with any available network, even with sometimes spotty wifi or mobile networks (library buses/bookmobiles). Nowadays we only have a couple of old sorting devices that still use stunnels because they were really expensive, hard to replace and they don't have an active software development team anymore to make necessary changes to them for SIPoHTTPS support. This is fine, since no changes to the SIP server logic was needed, but we need to remember to mind them when we make server side changes that might affect them.
Comment 30 Katrin Fischer 2023-12-12 10:45:21 UTC
(In reply to Lari Strand from comment #29)
> Lyngsoe systems and Mikroväylä added support for their self-check machines
> for our SIPoHTTPS plugin system here in Finland. We have hundreds of
> machines here that talk with our Koha SIP servers, including door systems,
> smart book cabinets, etc. The need for this came because of the near
> impossibility to set up and manage stunnels for all devices without
> problems. Now we can maintain and reconfigure our server side freely without
> affecting the self check machines or need for reconfiguration for the
> stunnels. The self-check machines also work with any available network, even
> with sometimes spotty wifi or mobile networks (library buses/bookmobiles).
> Nowadays we only have a couple of old sorting devices that still use
> stunnels because they were really expensive, hard to replace and they don't
> have an active software development team anymore to make necessary changes
> to them for SIPoHTTPS support. This is fine, since no changes to the SIP
> server logic was needed, but we need to remember to mind them when we make
> server side changes that might affect them.

That's really useful information - thx for sharing! If your plugin is publicly available I think it could make a real good addition to the wiki page for some more visibility. I think there are also libraries in Germany using Lyngsoe.
Comment 31 Kyle M Hall 2023-12-12 14:03:20 UTC
(In reply to Lari Strand from comment #29)
> Lyngsoe systems and Mikroväylä added support for their self-check machines
> for our SIPoHTTPS plugin system here in Finland. We have hundreds of
> machines here that talk with our Koha SIP servers, including door systems,
> smart book cabinets, etc. The need for this came because of the near
> impossibility to set up and manage stunnels for all devices without
> problems. Now we can maintain and reconfigure our server side freely without
> affecting the self check machines or need for reconfiguration for the
> stunnels. The self-check machines also work with any available network, even
> with sometimes spotty wifi or mobile networks (library buses/bookmobiles).
> Nowadays we only have a couple of old sorting devices that still use
> stunnels because they were really expensive, hard to replace and they don't
> have an active software development team anymore to make necessary changes
> to them for SIPoHTTPS support. This is fine, since no changes to the SIP
> server logic was needed, but we need to remember to mind them when we make
> server side changes that might affect them.

💯
Comment 32 David Cook 2023-12-12 22:41:09 UTC
(In reply to Lari Strand from comment #29)
> Lyngsoe systems and Mikroväylä added support for their self-check machines
> for our SIPoHTTPS plugin system here in Finland. We have hundreds of
> machines here that talk with our Koha SIP servers, including door systems,
> smart book cabinets, etc. The need for this came because of the near
> impossibility to set up and manage stunnels for all devices without
> problems. Now we can maintain and reconfigure our server side freely without
> affecting the self check machines or need for reconfiguration for the
> stunnels. 


When you say "manage stunnels for all devices", do you mean stunnel for the SIP client (e.g. client software in self-check machine)? Or do you mean stunnel for the Koha SIP server? Or both?

Setting up stunnel for the Koha SIP server should be very simple, but I could see how setting up stunnel for all the SIP clients could be challenging. But then if the self-check provider added support for SIP over HTTPS, then why couldn't they just add support for SIP over TLS? Surely that would've been easier as well.

In any case, it's cool that you've got it all working! I'm just keen to understand is all :)
Comment 33 Lari Strand 2023-12-13 07:42:24 UTC
> When you say "manage stunnels for all devices", do you mean stunnel for the
> SIP client (e.g. client software in self-check machine)? Or do you mean
> stunnel for the Koha SIP server? Or both?
> 
> Setting up stunnel for the Koha SIP server should be very simple, but I
> could see how setting up stunnel for all the SIP clients could be
> challenging. But then if the self-check provider added support for SIP over
> HTTPS, then why couldn't they just add support for SIP over TLS? Surely that
> would've been easier as well.
> 
> In any case, it's cool that you've got it all working! I'm just keen to
> understand is all :)

I haven't worked with the stunnels myself on our systems or even asked one to be set up, but as far as I understand it needs work from both our server's host/provider's support personnel (which is not us) + the municipalities' IT personnel for each library and they have to work together to set up a tunnel. We can only ask them to enable an stunnel to be set up between a self check machine's network and out server host. So it's out of our hands how they manage to set them up and in what time (+upkeep). So naturally we wanted to get rid of all that hassle.

I haven't looked at SIP over TLS so I have no experience of that. Does Koha's SIP server support it natively?
Comment 34 Lari Strand 2023-12-13 08:05:51 UTC
I googled a bit and seems This TLS thing is something our server service provider support personnel would have to set up with the self check machines' support people? Maybe too many extra steps for us compared to just pointing the self check machines etc. to a Koha SIPoHTTPS endpoint. Not sure though...
Comment 35 Lari Strand 2023-12-13 08:40:39 UTC
Seems like a valid alternative approach if the self check machines could support/could add support to their software.
Comment 36 David Cook 2023-12-13 23:41:20 UTC
(In reply to Lari Strand from comment #33)
> I haven't worked with the stunnels myself on our systems or even asked one
> to be set up, but as far as I understand it needs work from both our
> server's host/provider's support personnel (which is not us) + the
> municipalities' IT personnel for each library and they have to work together
> to set up a tunnel. We can only ask them to enable an stunnel to be set up
> between a self check machine's network and out server host. So it's out of
> our hands how they manage to set them up and in what time (+upkeep). So
> naturally we wanted to get rid of all that hassle.

If you don't have server access, then you will need a sysadmin with server access to setup the stunnel on the server, yes.

However, it's very easy. It's a few lines of configuration that you do 1 time. You only need to update the SSL certificate like you'd need to do with any SSL-based service (like a HTTPS listener on a web server).

As for the client side, yes that would be up to whoever is setting up the devices. It depends on their infrastructure. They could have 1 stunnel proxy on their local network which every self-check device connects to, which then proxies the connection (over SSL) to your server's stunnel. 

The client and server ends can be set up independently of each other, and then at the end an integration test can be done to make sure the client can talk to the server. It's pretty easy.

> I haven't looked at SIP over TLS so I have no experience of that. Does
> Koha's SIP server support it natively?

No, that's what I was saying before. Koha's SIP server does *not* support SSL/TLS natively, which is why stunnel is needed.

However, Koha's SIP server could be updated fairly easily to support it natively I think. 

(In reply to Lari Strand from comment #34)
> I googled a bit and seems This TLS thing is something our server service
> provider support personnel would have to set up with the self check
> machines' support people? Maybe too many extra steps for us compared to just
> pointing the self check machines etc. to a Koha SIPoHTTPS endpoint. Not sure
> though...

As I noted above, the client and server are separate. Your server service provider would set up the stunnel service on the server. This is like any server software. You can test it independently on your own. 

After the server is setup, the self check machines' support people would need to test to make sure they can connect, but that's a fairly normal procedure.

SIPoHTTPS is using TLS too, but you're managing the TLS in the Apache web server instead of the Stunnel proxy server.

(In reply to Lari Strand from comment #35)
> Seems like a valid alternative approach if the self check machines could
> support/could add support to their software.

So some self check machines have built-in SSL/TLS support. But many don't and that's why they use a stunnel client to proxy traffic over SSL/TLS. 

But if the self check machines you're talking about are adding SIPoHTTPS support, then they are adding SSL/TLS support... just via a HTTP client rather than a SIP client. So they're already doing the work...

But HTTP is a lot more well understood of an application layer protocol than SIP.
Comment 37 David Cook 2023-12-13 23:43:47 UTC
But in the end... whatever works :D
Comment 38 Lari Strand 2023-12-14 10:37:21 UTC
Oh ok thanks for the explanation.

These three major device manufactureres Mikroväylä, Lyngsoe systems and Bibliotheca all support SIPoHTTP and it has almost become a standard in that sense at least here. A parallel implementation that aims to do the same thing would be a bit overkill. If we ever intergated some of this stuff in Koha (not as a plugin but native support), we should pick one or the other? Or just leave both SIP over TLS (if doable) and SIPoHTTP implementations as plugins to cover local needs?
Comment 39 Lari Strand 2023-12-14 11:00:43 UTC
But as I said SIPoHTTP has been working extremely well for us and has been tried and tested with almost a thousand devices in our production environment for almost two years now and we have been very pleased with the performance, stability + ease of configuration (all aspects of the system can be set up and managed by us developers and librarians basically, without need for 3rd parties in between, although librarians usually send support tickets to the SIP devices' manufacturers support personnel for big batch changes for device configurations or when a librarian isn't tech-savvy :D ).
Comment 40 Lari Strand 2023-12-14 11:23:32 UTC
We have also been thinking of another enhancemet for SIPoHTTP. Getting rid of the need to configure the SIP server configs for new/changed device accounts and instead using koha objects + some extra code for the SIPoHTTP endpoint for authenticating the accounts and thus getting rid of the extra work us developers have to do, managing them server side in SIPconfigs. That way the work of managing the device accounts would be left to just the librarians to do in Koha.
Comment 41 David Cook 2023-12-14 23:31:00 UTC
(In reply to Lari Strand from comment #40)
> We have also been thinking of another enhancemet for SIPoHTTP. Getting rid
> of the need to configure the SIP server configs for new/changed device
> accounts and instead using koha objects + some extra code for the SIPoHTTP
> endpoint for authenticating the accounts and thus getting rid of the extra
> work us developers have to do, managing them server side in SIPconfigs. That
> way the work of managing the device accounts would be left to just the
> librarians to do in Koha.

That does sound like a useful enhancement. 

(In reply to Lari Strand from comment #38)
> Oh ok thanks for the explanation.
> 
> These three major device manufactureres Mikroväylä, Lyngsoe systems and
> Bibliotheca all support SIPoHTTP and it has almost become a standard in that
> sense at least here. A parallel implementation that aims to do the same
> thing would be a bit overkill. If we ever intergated some of this stuff in
> Koha (not as a plugin but native support), we should pick one or the other?
> Or just leave both SIP over TLS (if doable) and SIPoHTTP implementations as
> plugins to cover local needs?

In theory, there would be model code that handles all the core SIP logic, and the SIP server and SIPoHTTP endpoint would just be different controllers. Easier said than done though of course.

That's cool about it becoming a standard over there. I don't think the rest of the world is ready for SIPoHTTP, but maybe they will be one day.