The 'accounts' and 'institutions' sections of the SIP Server configuration should be able to be moved into the Koha Staff Client.. this would help librarians understand what settings are in place and allow them to make small changes to those configurations as required.
Created attachment 172530 [details] SIPconfig.xml into DB schema Hi all I've started looking into this. @Kyle @Nick, can I ask a few minutes of your time to briefly look at this schema before I fully commit to it. I aimed to catch and accommodate all possible configurations values from SIPconfig.xml. Thank you.
Created attachment 172584 [details] SIPconfig.xml into DB schema sip_institution should be a 1 to 1 with sip_policy, even though SIPconfig.xml technically allows for multiple policies per institution, the template SIPconfig.xml only has one policy per institution and as I understand it the SIP code logic expects only one policy per institution. This being the case, and policies not relating to anything else, there's no reason to have a separate table for policies specifically and these columns can be merged into sip_institutions instead. Thoughts?
(In reply to Pedro Amorim from comment #2) > Created attachment 172584 [details] > SIPconfig.xml into DB schema > > sip_institution should be a 1 to 1 with sip_policy, even though > SIPconfig.xml technically allows for multiple policies per institution, the > template SIPconfig.xml only has one policy per institution and as I > understand it the SIP code logic expects only one policy per institution. > This being the case, and policies not relating to anything else, there's no > reason to have a separate table for policies specifically and these columns > can be merged into sip_institutions instead. Thoughts? I agree with everything you wrote. I do note the image has institutions and accounts being one to one, but they should be one to many.
(In reply to Kyle M Hall (khall) from comment #3) > I do note the image has institutions and > accounts being one to one, but they should be one to many. 100%. Tyvm.
Hi all, this is ready for testing. I'm pasting here the same test plan that is present on the last commit of the branch, the migration test commit: Test plan for the whole module: 1) Checkout the branch and reset_all 2) Visit the administration page: /cgi-bin/koha/admin/admin-home.pl 3) Notice the new 'Self-service circulation (SIP2)' item under 'Patrons and circulation'. Click it. 4) Notice you are now on the new SIP2 module. Create some institutions, accounts, listeners, etc. Play around with the forms. 5) Notice everytime you add/edit a new SIP2 'resource', a prompt asking to restart the SIP2 server is shown. If you confirm it, it will restart the SIP server in order for the new data to take effect. 6) Run 'updatedatabase'. Notice the contents from /etc/koha/sites/kohadev/SIPConfig.xml have been migrated to the database are should be accessible in the UI. 7) Confirm all the data has been migrated correctly, and is editable in the UI as expected. 8) You can repeat this test plan but edit the contents of /etc/koha/sites/kohadev/SIPConfig.xml with a more complex and thorough configuration (something resembling a real SIPConfig.xml) 9) After updating the contents of SIPConfig.xml, restart SIP: koha-sip --restart 10) Test the module again, verify the contents have been migrated as expected. 11) Run the migration test again to ensure that SIP2 is being served the exact same configuration before and after the migration: prove t/db_dependent/Koha/SIP2/SIP2ModuleMigration.t
I'm getting error: DEV atomic update /home/lmstrand/Koha25/installer/data/mysql/atomicupdate/bug_37893.pl [14:58:31]: Bug 37893 - Move SIPconfig.xml to database ERROR: {UNKNOWN}: DBI Exception: DBD::mysql::db do failed: Can't create table `kohacomm`.`sip_accounts` (errno: 150 "Foreign key constraint is incorrectly formed") at /home/lmstrand/Koha25/C4/Installer.pm line 771
Seems the table sip_institutions needs to be generated first. We also have multiple sip config files on our server (under SIPconfig path) that should be iterated through and the file locations are on a different location where the atomicupdate is trying to find them. Maybe the path to the SIPConfig files could be set and read from koha-conf.xml or something?
Maybe the passwords could be obscured by ***** by default in the sip2 accounts view and could be shown only when a "view password" button is clicked or something for security reasons?
"We also have multiple sip config files on our server (under SIPconfig path) that should be iterated through" This could be difficult to do and managed via the UI, so as a workaround combining our config files into one and running just that sip server process would be an ok solution for us. Anyways I'm thrilled about this, good work!
Hi Lari, your feedback and testing is very much appreciated. Thank you for looking. (In reply to Lari Strand from comment #6) > I'm getting error: DEV atomic update > /home/lmstrand/Koha25/installer/data/mysql/atomicupdate/bug_37893.pl > [14:58:31]: Bug 37893 - Move SIPconfig.xml to database > ERROR: {UNKNOWN}: DBI Exception: DBD::mysql::db do failed: Can't create > table `kohacomm`.`sip_accounts` (errno: 150 "Foreign key constraint is > incorrectly formed") at /home/lmstrand/Koha25/C4/Installer.pm line 771 This was an oversight, I've corrected it (sip_institutions needs to be created before sip_accounts) and squashed to https://github.com/PTFS-Europe/koha/commit/c0937ae0b8644fd42c4b84687235407cfb29e73a (In reply to Lari Strand from comment #7) > We also have multiple sip config files on our server (under SIPconfig path) > that should be iterated through and the file locations are on a different > location where the atomicupdate is trying to find them. Maybe the path to > the SIPConfig files could be set and read from koha-conf.xml or something? This is an interesting use case, apologies for my ignorance but how exactly do you have this set up? i.e. how does Koha natively know that it needs to look for multiple files on a certain directory? (In reply to Lari Strand from comment #8) > Maybe the passwords could be obscured by ***** by default in the sip2 > accounts view and could be shown only when a "view password" button is > clicked or something for security reasons? I love this idea, this needs to be built into bug bug 38201 as a "password" type form field, on the framework, and then all modules would be able to use it, including this work here of course. (In reply to Lari Strand from comment #9) > "We also have multiple sip config files on our server (under SIPconfig path) > that should be iterated through" This could be difficult to do and managed > via the UI, so as a workaround combining our config files into one and > running just that sip server process would be an ok solution for us. Coming back to this, is your setup a custom thing or a Koha native thing? If Koha natively supports multiple SIPconfig.xml files then this work needs to consider that. I've not considered it because I'm not aware of it being possible but I'm not very knowledgeable in how SIP may or may not be configured. If this is something you have set up in a custom manner, then yes I think adapting your custom set up for when upgrading into this module is the best course of action.
We've been testing this with much success! We've successfully configured connections and run tests of most available options. We have a few things yet to confirm. Right up front, though, we're having trouble with the modal to restart SIP after editing the configuration. SIP did not restart when we selected Yes. Our changes did not save when we selected No. Either way, the modal kept coming back even though we checked the "Don't ask again" checkbox. Less pressingly, it'd be nice to get tooltips in the Institutions setup explaining what's wanted in each field.
Thank you so much Andrew for your reply and your time. I've taken note of this and there is indeed a problem in the SIP restart logic (it's not working). I've been meaning to come back to this but it's not been possible, yet! I'm aiming to follow-up here by end of this week or during hackfest next week. Thank you.
Hi all, hi Andrew. Thank you for your patience here. I've updated the branch to pick up on database changes without having to restart the SIP server. The latest commit as an additional test plan to test this specific functionality: Changes are picked up by SIP without restart: Test plan: 1) Using KTD, after having followed the original test plan, run the following SIP emulator command: $ perl misc/sip_cli_emulator.pl -su koha -sp koha -l kohalibrary --host localhost --patron 42 -m patron_information 2) Note that the response message contains 'Greetings from Koha'. 3) Using the module UI, visit SIP accounts: http://localhost:8081/cgi-bin/koha/sip2/accounts 4) Find the SIP account that has 'koha' as login. Click edit. 5) Scroll down to 'SIP response mappings' and expand 'Add new screen msg regex'. 6) On 'Find', enter 'Greetings from Koha'. On 'Replace', enter 'Hello from 37893 test plan'. Save. 7) Repeat the sip command from 1). Note that 'Greetings from Koha has been replaced with 'Hello from 37893 test plan', without having to restart the SIP server.
Failed QA At step 4 it is unclear what value should be set in implementation field. At step 6, nothing will import if you have created institutions, accounts, etc. Can we please have a more detailed test plan that covers all of the patches?
We think this is a really nice feature though!
I could see adding "accounts" and "institutions", but not "listeners". I suppose for a Koha system where they don't have sysadmin it would be handy if they could define "listeners", but for a multi-tenanted system you don't want Koha instances being able to define the listeners.
Test plan for the whole module: [MODULE UI TESTING] 1) Checkout the branch and reset_all 2) Run 'updatedatabase'. The contents from /etc/koha/sites/kohadev/SIPConfig.xml have been migrated to the database and should be accessible in the UI. 3) Visit the administration page: http://localhost:8081/cgi-bin/koha/admin/admin-home.pl 4) Notice the new 'Self-service circulation (SIP2)' item under 'Patrons and circulation'. Click it. 5) Notice you are now on the new SIP2 module. Create some institutions, accounts, listeners, etc. Play around with the forms. 6) Confirm all the data has been migrated correctly, and is editable in the UI as expected. [SIP TESTING] 1) On ktd, run the following SIP emulator command: $ perl misc/sip_cli_emulator.pl -su koha -sp koha -l kohalibrary --host localhost --patron 42 -m patron_information 2) Note that the response message contains 'Greetings from Koha'. 3) Using the module UI, visit SIP accounts: http://localhost:8081/cgi-bin/koha/sip2/accounts 4) Find the SIP account that has 'koha' as login. Click edit. 5) Scroll down to 'SIP response mappings' and click 'Add new screen msg regex'. 6) On 'Find', enter 'Greetings from Koha'. On 'Replace', enter 'Hello from 37893 test plan'. Save. 7) Repeat the sip command from 1). Note that 'Greetings from Koha has been replaced with 'Hello from 37893 test plan', without having to restart the SIP server. [OPTIONAL] 1) You can repeat the previous steps in [MODULE UI TESTING] but edit the contents of /etc/koha/sites/kohadev/SIPConfig.xml beforehand with a more complex and thorough configuration (something resembling a real SIPConfig.xml) 2) After updating the contents of SIPConfig.xml, restart SIP: koha-sip --restart 3) Test the module again, verify the contents have been migrated as expected.
(In reply to John Vinke from comment #14) > Failed QA Hi John, thank you. > At step 4 it is unclear what value should be set in implementation field. I pushed a new commit that should address this: https://github.com/openfifth/koha/commit/871ed139317ea56799909b5af8190b15470a02ee Please let me know. > At step 6, nothing will import if you have created institutions, accounts, > etc. > > Can we please have a more detailed test plan that covers all of the patches? I agree. I've made the previous 2 test plan comments obsolete to avoid causing future confusion. I've updated the test plan on the last tests patch on the branch and also copied here on my previous comment. The revised test plan should no longer cause you to experience this, i.e. first creating manually then import through updatedatabase. However, updatedatabase not adding subsequent data is intentional, as the data being re-inserted will fail due to duplicate keys, but the 'INSERT IGNORE INTO' skips the error. Furthermore, because updatedatabase is intended to run at upgrade time, it is not expected to have database data at that point, only data in the SIPconfig.xml file. If you want to run updatedatabase multiple times to test different setups or SIPconfig.xml files, the data should be removed beforehand to avoid duplicate clashes: delete from sip_accounts; delete from sip_institutions; delete from sip_listeners; delete from sip_server_params; delete from sip_system_preference_overrides;
(In reply to David Cook from comment #16) > I could see adding "accounts" and "institutions", but not "listeners". > > I suppose for a Koha system where they don't have sysadmin it would be handy > if they could define "listeners", but for a multi-tenanted system you don't > want Koha instances being able to define the listeners. The idea is to move XML file based configurations into a UI module. I agree that listeners are not quite the same as accounts or institutions, but the alternative would be to end up having half of SIP config in the UI module, and the other half in the XML file. The way this has been implemented is to fully move everything out of the XML file into the new UI module. I'm out of my depth when it comes to the inner workings of SIP, and the following may be a dumb question, but how does one know if a system is multi-tenanted or not? The UI configuration of listeners could be tied to this, i.e. only allow CRUD of listeners if not multi-tenanted.
> I'm out of my depth when it comes to the inner workings of SIP, and the following may be a dumb question, but how does one know if a system is multi-tenanted or not? The UI configuration of listeners could be tied to this, i.e. only allow CRUD of listeners if not multi-tenanted. I think multi-tenanted here means "more than one instance running on the same server". You don't want different instances to set up their SIPservers to listen on the same port. Not sure how to solve that in code, though. It might have to require people from the different instances to just talk to each other?
(In reply to Magnus Enger from comment #20) > I think multi-tenanted here means "more than one instance running on the > same server". You don't want different instances to set up their SIPservers > to listen on the same port. > > Not sure how to solve that in code, though. It might have to require people > from the different instances to just talk to each other? Multiple Koha instances, got it. The migration process is designed to pick up the correct respective SIPconfig.xml for the instance running the upgrade so it should only import it's respective data (including listeners). In relation to the issue of different listeners (from different Koha instances) using the same port, as far as I can tell this is not checked/enforced currently, i.e. this is already an issue/possibility before moving it into the module, though I agree that it may expose / make this more prone to happening since it'll become editable in the UI. I'm chatting with Martin about a possible solution we could try here.
Magnus and David, please take a look at this commit when you have the chance: https://github.com/openfifth/koha/commit/a0a003a5aaec12ac8103c3b4512478ddda1fbb8b
(In reply to Pedro Amorim from comment #21) > In relation to the issue of different listeners (from different Koha > instances) using the same port, as far as I can tell this is not > checked/enforced currently, i.e. this is already an issue/possibility before > moving it into the module, though I agree that it may expose / make this > more prone to happening since it'll become editable in the UI. (In reply to Pedro Amorim from comment #22) > Magnus and David, please take a look at this commit when you have the chance: > https://github.com/openfifth/koha/commit/ > a0a003a5aaec12ac8103c3b4512478ddda1fbb8b If you have 1 sysadmin managing N SIPconfig.xml, you have 1 person managing the IP addresses and ports across all the instances though, so conflicts aren't a problem. What I'm trying to say is that - as a vendor sysadmin - I don't want Koha library admins being able to configure the IP address and port number for the listener. There would be some potential for malicious abuse to the underlying host system if a user was able to dictate the IP address and port used. It would probably be somewhat limited since it would be running as the Koha user, so it wouldn't be able to bind to a privileged port, but it's still not great. Realistically, in production, the sysadmin will have likely set up stunnel and have that proxying to specific ports they've defined in SIPconfig.xml. There's also the case of the sysadmin or network admin setting up the firewalls. So I'm not quite sure of the benefit to moving the listener config into the web UI? I can only see the downsides.
(In reply to Pedro Amorim from comment #22) > Magnus and David, please take a look at this commit when you have the chance: > https://github.com/openfifth/koha/commit/ > a0a003a5aaec12ac8103c3b4512478ddda1fbb8b Looking at this again, I think you've got a typo for the $addr? You're trying to put the port into that variable? Even with that validation, the user would still be able to define any non-privileged port and any IPv4 address. (Note that the validation would prevent IPv6 addresses.) -- Just to reiterate... I'm not worried about web users being able to successfully bind to an IP address and port. I'm worried about them binding to addresses and ports that they shouldn't. (Of course, a workaround for that could be to have address and port limits in koha-conf.xml with some sensible defaults. Like a limit on ports 6000-6999 although limits on IP address could be trickier because of differing setups.)
This worked great. Andrew and I tested. As far as listeners go, we would have no problem if community wanted to not make listeners editable. It seems like the biggest place things could go wrong with this. It would be nice to see the information, but editing it could cause big headaches.
(In reply to Lisette Scheer from comment #25) > This worked great. Andrew and I tested. > As far as listeners go, we would have no problem if community wanted to not > make listeners editable. It seems like the biggest place things could go > wrong with this. It would be nice to see the information, but editing it > could cause big headaches. That's a good point about being able to see the information. It could be useful to let library admins see the information, so that they can have more informed discussions with their IT.