Summary: | Migrate some SIP configuration into the staff interface | ||
---|---|---|---|
Product: | Koha | Reporter: | Martin Renvoize (ashimema) <martin.renvoize> |
Component: | System Administration | Assignee: | Pedro Amorim <pedro.amorim> |
Status: | Failed QA --- | QA Contact: | Kyle M Hall (khall) <kyle> |
Severity: | new feature | ||
Priority: | P5 - low | CC: | andrew.auld, andrew, carthur, cbrannon, dcook, george, gmcharlt, jonathan.druart, jvinke, kyle, lisette, lmstrand, m.de.rooy, magnus, marion.durand, martin.renvoize, nick, pedro.amorim, rcoert, wizzyrea |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
URL: | https://github.com/openfifth/koha/commits/new_bug_37893 | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | Large patch | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: | Circulation function: | ||
Bug Depends on: | 38201, 39091 | ||
Bug Blocks: | 37248, 40120 | ||
Attachments: |
SIPconfig.xml into DB schema
SIPconfig.xml into DB schema |
Description
Martin Renvoize (ashimema)
2024-09-11 14:27:43 UTC
Created attachment 172530 [details]
SIPconfig.xml into DB schema
Hi all I've started looking into this.
@Kyle @Nick, can I ask a few minutes of your time to briefly look at this schema before I fully commit to it. I aimed to catch and accommodate all possible configurations values from SIPconfig.xml. Thank you.
Created attachment 172584 [details]
SIPconfig.xml into DB schema
sip_institution should be a 1 to 1 with sip_policy, even though SIPconfig.xml technically allows for multiple policies per institution, the template SIPconfig.xml only has one policy per institution and as I understand it the SIP code logic expects only one policy per institution.
This being the case, and policies not relating to anything else, there's no reason to have a separate table for policies specifically and these columns can be merged into sip_institutions instead. Thoughts?
(In reply to Pedro Amorim from comment #2) > Created attachment 172584 [details] > SIPconfig.xml into DB schema > > sip_institution should be a 1 to 1 with sip_policy, even though > SIPconfig.xml technically allows for multiple policies per institution, the > template SIPconfig.xml only has one policy per institution and as I > understand it the SIP code logic expects only one policy per institution. > This being the case, and policies not relating to anything else, there's no > reason to have a separate table for policies specifically and these columns > can be merged into sip_institutions instead. Thoughts? I agree with everything you wrote. I do note the image has institutions and accounts being one to one, but they should be one to many. (In reply to Kyle M Hall (khall) from comment #3) > I do note the image has institutions and > accounts being one to one, but they should be one to many. 100%. Tyvm. Hi all, this is ready for testing. I'm pasting here the same test plan that is present on the last commit of the branch, the migration test commit: Test plan for the whole module: 1) Checkout the branch and reset_all 2) Visit the administration page: /cgi-bin/koha/admin/admin-home.pl 3) Notice the new 'Self-service circulation (SIP2)' item under 'Patrons and circulation'. Click it. 4) Notice you are now on the new SIP2 module. Create some institutions, accounts, listeners, etc. Play around with the forms. 5) Notice everytime you add/edit a new SIP2 'resource', a prompt asking to restart the SIP2 server is shown. If you confirm it, it will restart the SIP server in order for the new data to take effect. 6) Run 'updatedatabase'. Notice the contents from /etc/koha/sites/kohadev/SIPConfig.xml have been migrated to the database are should be accessible in the UI. 7) Confirm all the data has been migrated correctly, and is editable in the UI as expected. 8) You can repeat this test plan but edit the contents of /etc/koha/sites/kohadev/SIPConfig.xml with a more complex and thorough configuration (something resembling a real SIPConfig.xml) 9) After updating the contents of SIPConfig.xml, restart SIP: koha-sip --restart 10) Test the module again, verify the contents have been migrated as expected. 11) Run the migration test again to ensure that SIP2 is being served the exact same configuration before and after the migration: prove t/db_dependent/Koha/SIP2/SIP2ModuleMigration.t I'm getting error: DEV atomic update /home/lmstrand/Koha25/installer/data/mysql/atomicupdate/bug_37893.pl [14:58:31]: Bug 37893 - Move SIPconfig.xml to database ERROR: {UNKNOWN}: DBI Exception: DBD::mysql::db do failed: Can't create table `kohacomm`.`sip_accounts` (errno: 150 "Foreign key constraint is incorrectly formed") at /home/lmstrand/Koha25/C4/Installer.pm line 771 Seems the table sip_institutions needs to be generated first. We also have multiple sip config files on our server (under SIPconfig path) that should be iterated through and the file locations are on a different location where the atomicupdate is trying to find them. Maybe the path to the SIPConfig files could be set and read from koha-conf.xml or something? Maybe the passwords could be obscured by ***** by default in the sip2 accounts view and could be shown only when a "view password" button is clicked or something for security reasons? "We also have multiple sip config files on our server (under SIPconfig path) that should be iterated through" This could be difficult to do and managed via the UI, so as a workaround combining our config files into one and running just that sip server process would be an ok solution for us. Anyways I'm thrilled about this, good work! Hi Lari, your feedback and testing is very much appreciated. Thank you for looking. (In reply to Lari Strand from comment #6) > I'm getting error: DEV atomic update > /home/lmstrand/Koha25/installer/data/mysql/atomicupdate/bug_37893.pl > [14:58:31]: Bug 37893 - Move SIPconfig.xml to database > ERROR: {UNKNOWN}: DBI Exception: DBD::mysql::db do failed: Can't create > table `kohacomm`.`sip_accounts` (errno: 150 "Foreign key constraint is > incorrectly formed") at /home/lmstrand/Koha25/C4/Installer.pm line 771 This was an oversight, I've corrected it (sip_institutions needs to be created before sip_accounts) and squashed to https://github.com/PTFS-Europe/koha/commit/c0937ae0b8644fd42c4b84687235407cfb29e73a (In reply to Lari Strand from comment #7) > We also have multiple sip config files on our server (under SIPconfig path) > that should be iterated through and the file locations are on a different > location where the atomicupdate is trying to find them. Maybe the path to > the SIPConfig files could be set and read from koha-conf.xml or something? This is an interesting use case, apologies for my ignorance but how exactly do you have this set up? i.e. how does Koha natively know that it needs to look for multiple files on a certain directory? (In reply to Lari Strand from comment #8) > Maybe the passwords could be obscured by ***** by default in the sip2 > accounts view and could be shown only when a "view password" button is > clicked or something for security reasons? I love this idea, this needs to be built into bug bug 38201 as a "password" type form field, on the framework, and then all modules would be able to use it, including this work here of course. (In reply to Lari Strand from comment #9) > "We also have multiple sip config files on our server (under SIPconfig path) > that should be iterated through" This could be difficult to do and managed > via the UI, so as a workaround combining our config files into one and > running just that sip server process would be an ok solution for us. Coming back to this, is your setup a custom thing or a Koha native thing? If Koha natively supports multiple SIPconfig.xml files then this work needs to consider that. I've not considered it because I'm not aware of it being possible but I'm not very knowledgeable in how SIP may or may not be configured. If this is something you have set up in a custom manner, then yes I think adapting your custom set up for when upgrading into this module is the best course of action. We've been testing this with much success! We've successfully configured connections and run tests of most available options. We have a few things yet to confirm. Right up front, though, we're having trouble with the modal to restart SIP after editing the configuration. SIP did not restart when we selected Yes. Our changes did not save when we selected No. Either way, the modal kept coming back even though we checked the "Don't ask again" checkbox. Less pressingly, it'd be nice to get tooltips in the Institutions setup explaining what's wanted in each field. Thank you so much Andrew for your reply and your time. I've taken note of this and there is indeed a problem in the SIP restart logic (it's not working). I've been meaning to come back to this but it's not been possible, yet! I'm aiming to follow-up here by end of this week or during hackfest next week. Thank you. Hi all, hi Andrew. Thank you for your patience here. I've updated the branch to pick up on database changes without having to restart the SIP server. The latest commit as an additional test plan to test this specific functionality: Changes are picked up by SIP without restart: Test plan: 1) Using KTD, after having followed the original test plan, run the following SIP emulator command: $ perl misc/sip_cli_emulator.pl -su koha -sp koha -l kohalibrary --host localhost --patron 42 -m patron_information 2) Note that the response message contains 'Greetings from Koha'. 3) Using the module UI, visit SIP accounts: http://localhost:8081/cgi-bin/koha/sip2/accounts 4) Find the SIP account that has 'koha' as login. Click edit. 5) Scroll down to 'SIP response mappings' and expand 'Add new screen msg regex'. 6) On 'Find', enter 'Greetings from Koha'. On 'Replace', enter 'Hello from 37893 test plan'. Save. 7) Repeat the sip command from 1). Note that 'Greetings from Koha has been replaced with 'Hello from 37893 test plan', without having to restart the SIP server. Failed QA At step 4 it is unclear what value should be set in implementation field. At step 6, nothing will import if you have created institutions, accounts, etc. Can we please have a more detailed test plan that covers all of the patches? We think this is a really nice feature though! I could see adding "accounts" and "institutions", but not "listeners". I suppose for a Koha system where they don't have sysadmin it would be handy if they could define "listeners", but for a multi-tenanted system you don't want Koha instances being able to define the listeners. Test plan for the whole module: [MODULE UI TESTING] 1) Checkout the branch and reset_all 2) Run 'updatedatabase'. The contents from /etc/koha/sites/kohadev/SIPConfig.xml have been migrated to the database and should be accessible in the UI. 3) Visit the administration page: http://localhost:8081/cgi-bin/koha/admin/admin-home.pl 4) Notice the new 'Self-service circulation (SIP2)' item under 'Patrons and circulation'. Click it. 5) Notice you are now on the new SIP2 module. Create some institutions, accounts, listeners, etc. Play around with the forms. 6) Confirm all the data has been migrated correctly, and is editable in the UI as expected. [SIP TESTING] 1) On ktd, run the following SIP emulator command: $ perl misc/sip_cli_emulator.pl -su koha -sp koha -l kohalibrary --host localhost --patron 42 -m patron_information 2) Note that the response message contains 'Greetings from Koha'. 3) Using the module UI, visit SIP accounts: http://localhost:8081/cgi-bin/koha/sip2/accounts 4) Find the SIP account that has 'koha' as login. Click edit. 5) Scroll down to 'SIP response mappings' and click 'Add new screen msg regex'. 6) On 'Find', enter 'Greetings from Koha'. On 'Replace', enter 'Hello from 37893 test plan'. Save. 7) Repeat the sip command from 1). Note that 'Greetings from Koha has been replaced with 'Hello from 37893 test plan', without having to restart the SIP server. [OPTIONAL] 1) You can repeat the previous steps in [MODULE UI TESTING] but edit the contents of /etc/koha/sites/kohadev/SIPConfig.xml beforehand with a more complex and thorough configuration (something resembling a real SIPConfig.xml) 2) After updating the contents of SIPConfig.xml, restart SIP: koha-sip --restart 3) Test the module again, verify the contents have been migrated as expected. (In reply to John Vinke from comment #14) > Failed QA Hi John, thank you. > At step 4 it is unclear what value should be set in implementation field. I pushed a new commit that should address this: https://github.com/openfifth/koha/commit/871ed139317ea56799909b5af8190b15470a02ee Please let me know. > At step 6, nothing will import if you have created institutions, accounts, > etc. > > Can we please have a more detailed test plan that covers all of the patches? I agree. I've made the previous 2 test plan comments obsolete to avoid causing future confusion. I've updated the test plan on the last tests patch on the branch and also copied here on my previous comment. The revised test plan should no longer cause you to experience this, i.e. first creating manually then import through updatedatabase. However, updatedatabase not adding subsequent data is intentional, as the data being re-inserted will fail due to duplicate keys, but the 'INSERT IGNORE INTO' skips the error. Furthermore, because updatedatabase is intended to run at upgrade time, it is not expected to have database data at that point, only data in the SIPconfig.xml file. If you want to run updatedatabase multiple times to test different setups or SIPconfig.xml files, the data should be removed beforehand to avoid duplicate clashes: delete from sip_accounts; delete from sip_institutions; delete from sip_listeners; delete from sip_server_params; delete from sip_system_preference_overrides; (In reply to David Cook from comment #16) > I could see adding "accounts" and "institutions", but not "listeners". > > I suppose for a Koha system where they don't have sysadmin it would be handy > if they could define "listeners", but for a multi-tenanted system you don't > want Koha instances being able to define the listeners. The idea is to move XML file based configurations into a UI module. I agree that listeners are not quite the same as accounts or institutions, but the alternative would be to end up having half of SIP config in the UI module, and the other half in the XML file. The way this has been implemented is to fully move everything out of the XML file into the new UI module. I'm out of my depth when it comes to the inner workings of SIP, and the following may be a dumb question, but how does one know if a system is multi-tenanted or not? The UI configuration of listeners could be tied to this, i.e. only allow CRUD of listeners if not multi-tenanted. > I'm out of my depth when it comes to the inner workings of SIP, and the following may be a dumb question, but how does one know if a system is multi-tenanted or not? The UI configuration of listeners could be tied to this, i.e. only allow CRUD of listeners if not multi-tenanted.
I think multi-tenanted here means "more than one instance running on the same server". You don't want different instances to set up their SIPservers to listen on the same port.
Not sure how to solve that in code, though. It might have to require people from the different instances to just talk to each other?
(In reply to Magnus Enger from comment #20) > I think multi-tenanted here means "more than one instance running on the > same server". You don't want different instances to set up their SIPservers > to listen on the same port. > > Not sure how to solve that in code, though. It might have to require people > from the different instances to just talk to each other? Multiple Koha instances, got it. The migration process is designed to pick up the correct respective SIPconfig.xml for the instance running the upgrade so it should only import it's respective data (including listeners). In relation to the issue of different listeners (from different Koha instances) using the same port, as far as I can tell this is not checked/enforced currently, i.e. this is already an issue/possibility before moving it into the module, though I agree that it may expose / make this more prone to happening since it'll become editable in the UI. I'm chatting with Martin about a possible solution we could try here. Magnus and David, please take a look at this commit when you have the chance: https://github.com/openfifth/koha/commit/a0a003a5aaec12ac8103c3b4512478ddda1fbb8b (In reply to Pedro Amorim from comment #21) > In relation to the issue of different listeners (from different Koha > instances) using the same port, as far as I can tell this is not > checked/enforced currently, i.e. this is already an issue/possibility before > moving it into the module, though I agree that it may expose / make this > more prone to happening since it'll become editable in the UI. (In reply to Pedro Amorim from comment #22) > Magnus and David, please take a look at this commit when you have the chance: > https://github.com/openfifth/koha/commit/ > a0a003a5aaec12ac8103c3b4512478ddda1fbb8b If you have 1 sysadmin managing N SIPconfig.xml, you have 1 person managing the IP addresses and ports across all the instances though, so conflicts aren't a problem. What I'm trying to say is that - as a vendor sysadmin - I don't want Koha library admins being able to configure the IP address and port number for the listener. There would be some potential for malicious abuse to the underlying host system if a user was able to dictate the IP address and port used. It would probably be somewhat limited since it would be running as the Koha user, so it wouldn't be able to bind to a privileged port, but it's still not great. Realistically, in production, the sysadmin will have likely set up stunnel and have that proxying to specific ports they've defined in SIPconfig.xml. There's also the case of the sysadmin or network admin setting up the firewalls. So I'm not quite sure of the benefit to moving the listener config into the web UI? I can only see the downsides. (In reply to Pedro Amorim from comment #22) > Magnus and David, please take a look at this commit when you have the chance: > https://github.com/openfifth/koha/commit/ > a0a003a5aaec12ac8103c3b4512478ddda1fbb8b Looking at this again, I think you've got a typo for the $addr? You're trying to put the port into that variable? Even with that validation, the user would still be able to define any non-privileged port and any IPv4 address. (Note that the validation would prevent IPv6 addresses.) -- Just to reiterate... I'm not worried about web users being able to successfully bind to an IP address and port. I'm worried about them binding to addresses and ports that they shouldn't. (Of course, a workaround for that could be to have address and port limits in koha-conf.xml with some sensible defaults. Like a limit on ports 6000-6999 although limits on IP address could be trickier because of differing setups.) This worked great. Andrew and I tested. As far as listeners go, we would have no problem if community wanted to not make listeners editable. It seems like the biggest place things could go wrong with this. It would be nice to see the information, but editing it could cause big headaches. (In reply to Lisette Scheer from comment #25) > This worked great. Andrew and I tested. > As far as listeners go, we would have no problem if community wanted to not > make listeners editable. It seems like the biggest place things could go > wrong with this. It would be nice to see the information, but editing it > could cause big headaches. That's a good point about being able to see the information. It could be useful to let library admins see the information, so that they can have more informed discussions with their IT. *** Bug 33681 has been marked as a duplicate of this bug. *** Tested this successfully and I'm signing off on it. I'm still not keen on this idea, but looking at the new_bug_37893 branch at a glance... it looks like you can still use the existing SIPconfig.xml and I could just locally comment out all the Koha::SIP2::* sections in C4::SIP::Sip::Configuration... I think I am going to mark as Failed QA though, because there's no sanity checking on the Koha::SIP2::ServerParam object. Theoretically, you could DOS a server (accidentally or maliciously) by feeding in server-params that fill up the process table. I agree with the original idea that "accounts" and "institutions" should be moved into the DB/web UI, but "listeners" and "server-params" really should not be. "syspref_overrides"... I'm not familiar enough with the use of that in the SIP server, so I don't have ready comment on that one. (In reply to David Cook from comment #29) > Theoretically, you could DOS > a server (accidentally or maliciously) by feeding in server-params that fill > up the process table. Can you please provide an example / test plan to reproduce / replicate this so I can address it? (In reply to David Cook from comment #30) > I agree with the original idea that "accounts" and "institutions" should be > moved into the DB/web UI, but "listeners" and "server-params" really should > not be. > > "syspref_overrides"... I'm not familiar enough with the use of that in the > SIP server, so I don't have ready comment on that one. You are proposing that only the accounts and institutions part is moved out of SIPConfig.xml. Everything else related to SIP configuration is kept in SIPConfig.xml. Is that correct? @David Cook, would a new 'sip2_module' staff permission here help / be enough? I think that syspref_override car also be migrated in the interface Sorry my bug mail is broken so I didn't get this. Just noticed your response by chance... (In reply to Pedro Amorim from comment #31) > (In reply to David Cook from comment #29) > > Theoretically, you could DOS > > a server (accidentally or maliciously) by feeding in server-params that fill > > up the process table. > > Can you please provide an example / test plan to reproduce / replicate this > so I can address it? Just looking up the process limits now. It looks like filling up the process table might not be possible when running the SIP server as a non-root user, since non-root users should have limits on the number of processes they can run. So it looks like I might be wrong there. But you could still create problems even as a non-root user. Basically just put the your min_servers, max_servers, and min_spare_servers to very high values. For instance, max_servers 500000 and min_spare_servers 450000. Keep in mind that the server-params are passed to the SIP server completely un-checked and un-sanitized, and that all parameters listened in the "CONFIGURATION FILE" section of Net::Server::PreFork can be used. It's just way too much control. > (In reply to David Cook from comment #30) > > I agree with the original idea that "accounts" and "institutions" should be > > moved into the DB/web UI, but "listeners" and "server-params" really should > > not be. > > > > "syspref_overrides"... I'm not familiar enough with the use of that in the > > SIP server, so I don't have ready comment on that one. > > You are proposing that only the accounts and institutions part is moved out > of SIPConfig.xml. Everything else related to SIP configuration is kept in > SIPConfig.xml. Is that correct? More or less. Basically, I'm saying "listeners" and "server-params" should stay in SIPConfig.xml. (In reply to Pedro Amorim from comment #32) > @David Cook, would a new 'sip2_module' staff permission here help / be > enough? Nope, because these are server-level parameters and not Koha/application-level parameters. The only web user permission that could potentially make sense for this is if we created a new user category like "system_administrator" and not give anyone that category except the system administrator that actually manages that server. These are not things that libraries should be able to manage themselves. Just the person managing the server. (In reply to David Cook from comment #34) > table might not be possible when running the SIP server as a non-root user, > since non-root users should have limits on the number of processes they can > run. So it looks like I might be wrong there. > > But you could still create problems even as a non-root user. > > Basically just put the your min_servers, max_servers, and min_spare_servers > to very high values. For instance, max_servers 500000 and min_spare_servers > 450000. If you wanted to check for the DOS, I'd say up those more to 5000000 for max_servers and 4500000 for min_spare_servers. You might need "min_servers" to match "min_spare_servers". I can't quite recall. (Taking a quick look at the code and yeah it looks like min_serves must be more than min_spare_servers at least.) Overall, just try to break stuff. Set log_level to 4 and change log_file to /tmp/hack_file.log or something else writeable by the Koha user (like /etc/koha/sites/<koha_instance>/hack_file.log) and try to fill up the file system and cause crashes that way. (Apologies if it takes me some time to respond. My mail server is blocking Bugzilla so I'm not getting email notifications at this time.) *** Bug 21935 has been marked as a duplicate of this bug. *** Just an idea here (but it might be managed in a separate bug after this one is pushed), it would be amazing to be able to duplicate all translation (screen_msg_regx) from an account to another. And even more to export and import all translation for transfer between Koha. It is currently quite difficult to manage translation when maintaining multiple instances which all have multiple login. (In reply to Marion Durand from comment #39) > Just an idea here (but it might be managed in a separate bug after this one > is pushed), it would be amazing to be able to duplicate all translation > (screen_msg_regx) from an account to another. And even more to export and > import all translation for transfer between Koha. > > It is currently quite difficult to manage translation when maintaining > multiple instances which all have multiple login. Would a 'duplicate' (full resource) feature address this? The idea being you'd duplicate an existing SIP2 account (create a new one with values from previous already filled in), keep the things you want (e.g. screen_msg_regx) and remove/change the rest? I ask this because bug 38201 (that blocks this one) aims to move development towards a more framework oriented approach. If this 'duplicate' feature is developed at the framework level, every VueJS module would be able to make use of it (including this SIP2 module), as this 'duplicate' functionality would certainly be useful in other modules / use cases (?) (In reply to Pedro Amorim from comment #40) > (In reply to Marion Durand from comment #39) > > Just an idea here (but it might be managed in a separate bug after this one > > is pushed), it would be amazing to be able to duplicate all translation > > (screen_msg_regx) from an account to another. And even more to export and > > import all translation for transfer between Koha. > > > > It is currently quite difficult to manage translation when maintaining > > multiple instances which all have multiple login. > > Would a 'duplicate' (full resource) feature address this? The idea being > you'd duplicate an existing SIP2 account (create a new one with values from > previous already filled in), keep the things you want (e.g. screen_msg_regx) > and remove/change the rest? Yes it would definitely be a great help ! Even if a way to export+import screen messages would still be nice as it allows sharing translation across different libraries and instances of Koha. |