The koha-stop-zebra command can finish without ensuring that zebrasrv is actually stopped. In particular, if there is an active query running when daemon --stop is run, the following state of affairs will occur until the query finishes or times out: - the child zebrasrv processing the query continues to run - the parent zebrasrv process becomes a zombie - the daemon(1) process continues to run -- and doesn't delete the lock file until the children go away. This is particularly problematic when a koha-stop-zebra is followed by a koha-start-zebra in quick succession, as happens during a logrotate. This is because while zebrasrv is in the half-stopped, half-started stop, the lock file still exists... and koha-start-zebra will see it and believe that zebrasrv is still running. When the query holding the child zebrasrv process finishes, the instance is left with no running Zebra.
I think this explains a case where I had a stopped zebrasrv on the day the logs rotated.
Sounds exactly like what we are experiencing.
From Philippe Blouin on the dev ml: === QUOTE We've had this problem on 10% of our installations. I solved it by going nuclear with logrotate since that was really hurting the users to not have zebra working (it would appear like working on our monitor, but was actually just dying and restarting). Here is the postrotate I've end up with, after many iterations of Next Level Apocalypse You :) postrotate service apache2 reload find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" stop' {} \; sleep 3 find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" stop' {} \; sleep 3 pkill -9 zebra sleep 2 pkill -9 zebra sleep 2 find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" start' {} \; endscript As you see, no subtlety, but I've had no problem for a year now... == END QUOTE
Created attachment 53925 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch actually loops 10 tens to check, before giving up. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3
Created attachment 53926 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch actually loops 10 tens to check, before giving up. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan
Comment on attachment 53926 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv Review of attachment 53926 [details] [review]: ----------------------------------------------------------------- ::: debian/scripts/koha-stop-zebra @@ +35,5 @@ > + # both daemon and zebrasrv processes for one instance > + # will go away, but there could be other processes > + # for other instances. > + expecting=`ps aux | grep zebra | wc -l` > + expecting=$((expecting-2)) zebrasrv forks a process for every active connection, so one can't count on there being exactly two processes.
Per my previous comment, I don't think Mark's patch will be reliable. I think a better approach would be to use start-stop-daemon, a la the init script that ships with IndexData's own package of Zebra (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils.zebrasrv.init#L69): start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME Unfortunately, daemon(1) doesn't appear to have any ability to specify using SIGKILL.
(In reply to Galen Charlton from comment #7) > Per my previous comment, I don't think Mark's patch will be reliable. > > I think a better approach would be to use start-stop-daemon, a la the init > script that ships with IndexData's own package of Zebra > (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils. > zebrasrv.init#L69): > > start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE > --name $NAME > > Unfortunately, daemon(1) doesn't appear to have any ability to specify using > SIGKILL. I agree, and we already use start-stop-daemon koha-plack. Good catch, Galen!
Created attachment 53958 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan
Created attachment 54314 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Works as expected, no regressions found.
Created attachment 54373 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Works as expected, no regressions found.
Pushed to master for 16.11, thanks Mark!
Pushed in 16.05. Will be in 16.05.03.
Pushed to 3.22.x, will be in 3.22.10