| Summary: | Restarting koha-common fails to restart SIP2 server | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Joonas Kylmälä <joonas.kylmala> |
| Component: | Packaging | Assignee: | Bugs List <koha-bugs> |
| Status: | Signed Off --- | QA Contact: | Testopia <testopia> |
| Severity: | normal | ||
| Priority: | P5 - low | CC: | david, lari.taskula, mtj, nugged, wizzyrea |
| Version: | Main | ||
| Hardware: | All | ||
| OS: | All | ||
| GIT URL: | Initiative type: | --- | |
| Sponsorship status: | --- | Comma delimited list of Sponsors: | |
| Crowdfunding goal: | 0 | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: |
This fixes an issue when stopping and restarting SIP servers using `koha-sip` - it would sometimes not restart the SIP servers.
This was because a restart could attempt the --start command while a previous SIP server was still running. This could result in the SIP server not restarting at all.
The fix replaces `daemon --stop` with `start-stop-daemon --stop` in the code, to ensure that all the running SIP servers are actually stopped before restarting.
|
Version(s) released in: | |
| Circulation function: | |||
| Attachments: |
Bug 27115: Use start-stop-daemon to stop in koha-sip
Bug 27115: Use start-stop-daemon to stop in koha-sip |
||
|
Description
Joonas Kylmälä
2020-11-30 08:01:55 UTC
I am not sure if this is still valid, but at least there is no mention of SIP in the output of koha-common restart: sudo service koha-common restart Restarting Koha ILS: koha-commonStopping Plack daemon for kohadev:. Stopping Z39.50/SRU daemon for kohadev:. Stopping Koha worker daemon for kohadev (default):. Stopping Koha worker daemon for kohadev (long_tasks):. Stopping Koha indexing daemon for kohadev:. Successfully stopped all services for instance koha-common:. Starting Plack daemon for kohadev:. Starting Z39.50/SRU daemon for kohadev:. Starting Koha worker daemon for kohadev (default):. Starting Koha worker daemon for kohadev (long_tasks):. Starting Koha indexing daemon for kohadev:. . I've seen this too, I think it happens when there are a lot of SIP workers/processes and it takes some time for them to kill. This is still valid.
Upping verbosity for koha-sip --start reveals the underlying issue. The SIP daemon doesn't stop in time to be able to start again. During koha-common startup, we are checking if SIP is already running. If so, we will not start the SIP server, and it will stay in stopped state until it is manually started again.
See line 11 "SIP server already running for ...":
> 1 06:07:43.x systemd[1]: Stopping koha-common.service - Start required services for each Koha instance...
> 2 06:07:43.x koha-common[x]: * Stopping SIP server for production_instance
> 3 06:07:43.x koha-common[x]: ...done.
> 4 ...
> 5 06:07:44.x systemd[1]: koha-common.service: Deactivated successfully.
> 6 06:07:44.x systemd[1]: koha-common.service: Unit process x (daemon) remains running after unit stopped.
> 7 06:07:44.x systemd[1]: koha-common.service: Unit process x (es_indexer_daem) remains running after unit stopped.
> 8 06:07:44.x systemd[1]: Stopped koha-common.service - Start required services for each Koha instance
> 9 ...
> 10 06:07:44.x systemd[1]: Starting koha-common.service - Start required services for each Koha instance...
> 11 06:07:45.x koha-common[x]: * Warning: SIP server already running for production_instance
Created attachment 188707 [details] [review] Bug 27115: Use start-stop-daemon to stop in koha-sip There has been an issue where a restart of koha-sip would sometimes not bring the SIP server back alive. This is because we have not been waiting for koha-sip --stop to actually stop the SIP server(s). It is then possible for a restart to attempt the --start command while a previous instance of SIP server is still running. As the --start command in this case is not allowed, we would then end up not starting the SIP server(s) at all. This patch replaces daemon --stop with start-stop-daemon --stop to ensure koha-sip daemon actually stops. start-stop-daemon provides us the --retry option, which we are able to utilize in waiting for the stop process to finish. To test: 1. Before applying this patch 2. Run this command export SIPINSTANCE="your_koha_instance_name" You can your_koha_instance_name using the koha-list command 3. Configure a SIP server (add some min_servers, e.g. 6, to SIPConfig.xml) 4. koha-sip --enable $SIPINSTANCE 5. koha-sip --verbose --start $SIPINSTANCE 6. Observe "Starting SIP server for ..." 7. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 8. Observe "Stopping SIP server for instancename" 9. In the output, observe the number of SIP servers you defined in step 3, plus one. 10. koha-sip --verbose --start $SIPINSTANCE 11. Apply this patch, or edit the koha-sip directly to get changes in place in your environment, e.g. using command nano $(which koha-sip) 12. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 13. Observe "Stopping SIP server for instancename" 14. In the output, observe no more SIP servers alive (In reply to Lari Taskula from comment #3) > > 7 06:07:44.x systemd[1]: koha-common.service: Unit process x (es_indexer_daem) remains running after unit stopped. Btw this line also concerns me. A couple of months ago we had a case where ES indexer was suddenly no longer running, most likely after a restart. Could be related to this issue and perhaps we should use start-stop-daemon for koha-es-indexer --stop as well. Created attachment 190766 [details] [review] Bug 27115: Use start-stop-daemon to stop in koha-sip There has been an issue where a restart of koha-sip would sometimes not bring the SIP server back alive. This is because we have not been waiting for koha-sip --stop to actually stop the SIP server(s). It is then possible for a restart to attempt the --start command while a previous instance of SIP server is still running. As the --start command in this case is not allowed, we would then end up not starting the SIP server(s) at all. This patch replaces daemon --stop with start-stop-daemon --stop to ensure koha-sip daemon actually stops. start-stop-daemon provides us the --retry option, which we are able to utilize in waiting for the stop process to finish. To test: 1. Before applying this patch 2. Run this command export SIPINSTANCE="your_koha_instance_name" You can your_koha_instance_name using the koha-list command 3. Configure a SIP server (add some min_servers, e.g. 6, to SIPConfig.xml) 4. koha-sip --enable $SIPINSTANCE 5. koha-sip --verbose --start $SIPINSTANCE 6. Observe "Starting SIP server for ..." 7. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 8. Observe "Stopping SIP server for instancename" 9. In the output, observe the number of SIP servers you defined in step 3, plus one. 10. koha-sip --verbose --start $SIPINSTANCE 11. Apply this patch, or edit the koha-sip directly to get changes in place in your environment, e.g. using command nano $(which koha-sip) 12. koha-sip --verbose --stop $SIPINSTANCE; ps aux | grep $SIPINSTANCE-koha-sip 13. Observe "Stopping SIP server for instancename" 14. In the output, observe no more SIP servers alive Signed-off-by: David Nind <david@davidnind.com> I think I tested this correctly, and that I observed the expected results!
In KTD, the SIP server is already running, but I went through the process of stopping it and then disabling it before working through the test plan.
I've attempted a release note.
Testing notes (using KTD):
1. SIP is already running in KTD for kohadev:
sudo koha-sip --verbose --status kohadev
SIP server running for kohadev:.
2. Make the change in step 3 of the test plan - made min_servers = 6: vi /etc/koha/sites/kohadev/SIPconfig.xml
3. Stop the SIP server:
sudo koha-sip --verbose --stop kohadev
Stopping SIP server for kohadev:.
4. Disable the SIP server:
sudo koha-sip --verbose --disable kohadev
Stopping running SIP
Warning: SIP server not running for kohadev:.
Information: SIP server disabled for kohadev
5. Check that no SIP servers are running:
ps aux | grep kohadev-koha-sip
kohadev+ 17009 0.0 0.0 9008 2176 pts/0 S+ 18:07 0:00 grep kohadev-koha-sip
sudo koha-sip --verbose --status kohadev
SIP server not running for kohadev: failed!
6. Enable the SIP server:
sudo koha-sip --verbose --enable kohadev
Enabling SIP server for kohadev - edit /etc/koha/sites/kohadev/SIPconfig.xml to configure
7. Start the SIP server:
sudo koha-sip --verbose --start kohadev
Starting SIP server for kohadev:.
8. Check the number of servers running - this is 7 for me (min_servers = 6):
ps aux | grep $SIPINSTANCE-koha-sip
kohadev+ 17366 0.0 0.0 8176 1808 ? S 18:12 0:00 daemon --name=kohadev-koha-sip --errlog=/var/log/koha/kohadev/sip-error.log --output=/var/log/koha/kohadev/sip-output.log --verbose=1 --respawn --delay=30 --pidfiles=/var/run/koha/kohadev --user=kohadev-koha.kohadev-koha -- perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17368 0.8 0.6 308364 208892 ? S 18:12 0:01 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17373 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17374 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17375 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17376 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17377 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17378 0.0 0.5 308364 187492 ? S 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17517 0.0 0.0 9008 2304 pts/0 S+ 18:14 0:00 grep kohadev-koha-sip
9. Back to step 7 of the test plan:
sudo koha-sip --verbose --stop kohadev; ps aux | grep kohadev-koha-sip
Stopping SIP server for kohadev:.
kohadev+ 17366 0.0 0.0 8176 1808 ? S 18:12 0:00 daemon --name=kohadev-koha-sip --errlog=/var/log/koha/kohadev/sip-error.log --output=/var/log/koha/kohadev/sip-output.log --verbose=1 --respawn --delay=30 --pidfiles=/var/run/koha/kohadev --user=kohadev-koha.kohadev-koha -- perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17368 0.4 0.6 308364 208892 ? R 18:12 0:01 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17373 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17374 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17375 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17376 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17377 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17378 0.0 0.5 308364 188900 ? R 18:12 0:00 kohadev-koha-sip: perl /kohadevbox/koha/C4/SIP/SIPServer.pm /etc/koha/sites/kohadev/SIPconfig.xml
kohadev+ 17666 0.0 0.0 9008 2048 pts/0 S+ 18:16 0:00 grep kohadev-koha-sip
10. Step 10 of the test plan - start the SIP server (7 running after this):
sudo koha-sip --verbose --start kohadev
Starting SIP server for kohadev:.
11. After applying the patch - end result is 7 SIP servers running (from e ps aux command): reset_all + step 2 above + restart_all
12. Steps 12 to 14 of the test plan:
sudo koha-sip --verbose --stop kohadev; ps aux | grep kohadev-koha-sip
Stopping SIP server for kohadev:.
kohadev+ 20224 0.0 0.0 9008 2048 pts/0 S+ 18:28 0:00 grep kohadev-koha-sip
|