Bug 16885 - koha-stop-zebra should be more sure of stopping zebrasrv
Summary: koha-stop-zebra should be more sure of stopping zebrasrv
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Packaging (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: M. Tompsett
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-07-08 19:31 UTC by Galen Charlton
Modified: 2017-12-07 22:18 UTC (History)
12 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv (2.59 KB, patch)
2016-08-03 03:35 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv (2.62 KB, patch)
2016-08-03 04:13 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv (2.03 KB, patch)
2016-08-03 15:27 UTC, M. Tompsett
Details | Diff | Splinter Review
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv (2.13 KB, patch)
2016-08-11 14:39 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv (2.13 KB, patch)
2016-08-12 12:06 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Galen Charlton 2016-07-08 19:31:19 UTC
The koha-stop-zebra command can finish without ensuring that zebrasrv is actually stopped.  In particular, if there is an active query running when daemon --stop is run, the following state of affairs will occur until the query finishes or times out:

- the child zebrasrv processing the query continues to run
- the parent zebrasrv process becomes a zombie
- the daemon(1) process continues to run -- and doesn't delete the lock file until the children go away.

This is particularly problematic when a koha-stop-zebra is followed by a koha-start-zebra in quick succession, as happens during a logrotate. This is because while zebrasrv is in the half-stopped, half-started stop, the lock file still exists... and koha-start-zebra will see it and believe that zebrasrv is still running. When the query holding the child zebrasrv process finishes, the instance is left with no running Zebra.
Comment 1 Marcel de Rooy 2016-07-10 15:15:40 UTC
I think this explains a case where I had a stopped zebrasrv on the day the logs rotated.
Comment 2 M. Tompsett 2016-07-22 00:59:50 UTC
Sounds exactly like what we are experiencing.
Comment 3 Marcel de Rooy 2016-08-02 15:02:11 UTC
From Philippe Blouin on the dev ml:
=== QUOTE
We've had this problem on 10% of our installations.  I solved it by 
 going nuclear with logrotate since that was really hurting the users to 
 not have zebra working (it would appear like working on our monitor, but 
 was actually just dying and restarting).

 Here is the postrotate I've end up with, after many iterations of Next 
 Level Apocalypse You :)

      postrotate
          service apache2 reload
          find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service 
 "$(basename "$0")" stop' {} \;
          sleep 3
          find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service 
 "$(basename "$0")" stop' {} \;
          sleep 3
          pkill -9 zebra
          sleep 2
          pkill -9 zebra
          sleep 2
          find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service 
 "$(basename "$0")" start' {} \;
      endscript

 As you see, no subtlety, but I've had no problem for a year now...
== END QUOTE
Comment 4 M. Tompsett 2016-08-03 03:35:40 UTC Comment hidden (obsolete)
Comment 5 M. Tompsett 2016-08-03 04:13:33 UTC Comment hidden (obsolete)
Comment 6 Galen Charlton 2016-08-03 13:31:02 UTC
Comment on attachment 53926 [details] [review]
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv

Review of attachment 53926 [details] [review]:
-----------------------------------------------------------------

::: debian/scripts/koha-stop-zebra
@@ +35,5 @@
> +    # both daemon and zebrasrv processes for one instance
> +    # will go away, but there could be other processes
> +    # for other instances.
> +    expecting=`ps aux | grep zebra | wc -l`
> +    expecting=$((expecting-2))

zebrasrv forks a process for every active connection, so one can't count on there being exactly two processes.
Comment 7 Galen Charlton 2016-08-03 13:37:01 UTC
Per my previous comment, I don't think Mark's patch will be reliable.

I think a better approach would be to use start-stop-daemon, a la the init script that ships with IndexData's own package of Zebra (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils.zebrasrv.init#L69):

  start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME

Unfortunately, daemon(1) doesn't appear to have any ability to specify using SIGKILL.
Comment 8 Tomás Cohen Arazi 2016-08-03 14:22:53 UTC
(In reply to Galen Charlton from comment #7)
> Per my previous comment, I don't think Mark's patch will be reliable.
> 
> I think a better approach would be to use start-stop-daemon, a la the init
> script that ships with IndexData's own package of Zebra
> (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils.
> zebrasrv.init#L69):
> 
>   start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE
> --name $NAME
> 
> Unfortunately, daemon(1) doesn't appear to have any ability to specify using
> SIGKILL.

I agree, and we already use start-stop-daemon koha-plack.

Good catch, Galen!
Comment 9 M. Tompsett 2016-08-03 15:27:21 UTC Comment hidden (obsolete)
Comment 10 Tomás Cohen Arazi 2016-08-11 14:39:58 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2016-08-12 12:06:31 UTC
Created attachment 54373 [details] [review]
Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv

This patch follows Galen's suggestion in comment #7.

TEST PLAN
---------
1) Back up your koha logs as desired.
2) add something to /var/log/koha/{instance name}/intranet-error.log
3) ps aux | grep zebra
4) logrotate -f /etc/logrotate.d/koha-common
5) ps aux | grep zebra
   -- the zebrasrv and daemon process for zebra indexing
      didn't restart.
6) apply this patch against /usr/sbin/koha-stop-zebra
7) sudo koha-start-zebra {instance name}
8) ps aux | grep zebra
   -- the processes should have started up again.
9) add different junk to /var/log/koha/{instance name}/intranet-error.log
10) ps aux | grep zebra
11) logrotate -f /etc/logrotate.d/koha-common
12) ps aux | grep zebra
    -- the process ids for the zebrasrv and daemon processes should
       be different, but the number of processes is the same as before.
13) sign off, because its less ugly than comment #3

Sponsored-by: Tulong Aklatan

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Works as expected, no regressions found.
Comment 12 Kyle M Hall 2016-08-17 17:57:21 UTC
Pushed to master for 16.11, thanks Mark!
Comment 13 Frédéric Demians 2016-08-23 10:22:54 UTC
Pushed in 16.05. Will be in 16.05.03.
Comment 14 Julian Maurice 2016-08-24 10:10:03 UTC
Pushed to 3.22.x, will be in 3.22.10