Description
Galen Charlton
2016-07-08 19:31:19 UTC
I think this explains a case where I had a stopped zebrasrv on the day the logs rotated. Sounds exactly like what we are experiencing. From Philippe Blouin on the dev ml: === QUOTE We've had this problem on 10% of our installations. I solved it by going nuclear with logrotate since that was really hurting the users to not have zebra working (it would appear like working on our monitor, but was actually just dying and restarting). Here is the postrotate I've end up with, after many iterations of Next Level Apocalypse You :) postrotate service apache2 reload find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" stop' {} \; sleep 3 find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" stop' {} \; sleep 3 pkill -9 zebra sleep 2 pkill -9 zebra sleep 2 find /etc/init.d/ -name "*zebra-daemon" -exec bash -c 'service "$(basename "$0")" start' {} \; endscript As you see, no subtlety, but I've had no problem for a year now... == END QUOTE Created attachment 53925 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch actually loops 10 tens to check, before giving up. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Created attachment 53926 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch actually loops 10 tens to check, before giving up. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Comment on attachment 53926 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv Review of attachment 53926 [details] [review]: ----------------------------------------------------------------- ::: debian/scripts/koha-stop-zebra @@ +35,5 @@ > + # both daemon and zebrasrv processes for one instance > + # will go away, but there could be other processes > + # for other instances. > + expecting=`ps aux | grep zebra | wc -l` > + expecting=$((expecting-2)) zebrasrv forks a process for every active connection, so one can't count on there being exactly two processes. Per my previous comment, I don't think Mark's patch will be reliable. I think a better approach would be to use start-stop-daemon, a la the init script that ships with IndexData's own package of Zebra (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils.zebrasrv.init#L69): start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME Unfortunately, daemon(1) doesn't appear to have any ability to specify using SIGKILL. (In reply to Galen Charlton from comment #7) > Per my previous comment, I don't think Mark's patch will be reliable. > > I think a better approach would be to use start-stop-daemon, a la the init > script that ships with IndexData's own package of Zebra > (https://github.com/indexdata/idzebra/blob/master/debian/idzebra-2.0-utils. > zebrasrv.init#L69): > > start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE > --name $NAME > > Unfortunately, daemon(1) doesn't appear to have any ability to specify using > SIGKILL. I agree, and we already use start-stop-daemon koha-plack. Good catch, Galen! Created attachment 53958 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Created attachment 54314 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Works as expected, no regressions found. Created attachment 54373 [details] [review] Bug 16885: koha-stop-zebra should be more sure of stopping zebrasrv This patch follows Galen's suggestion in comment #7. TEST PLAN --------- 1) Back up your koha logs as desired. 2) add something to /var/log/koha/{instance name}/intranet-error.log 3) ps aux | grep zebra 4) logrotate -f /etc/logrotate.d/koha-common 5) ps aux | grep zebra -- the zebrasrv and daemon process for zebra indexing didn't restart. 6) apply this patch against /usr/sbin/koha-stop-zebra 7) sudo koha-start-zebra {instance name} 8) ps aux | grep zebra -- the processes should have started up again. 9) add different junk to /var/log/koha/{instance name}/intranet-error.log 10) ps aux | grep zebra 11) logrotate -f /etc/logrotate.d/koha-common 12) ps aux | grep zebra -- the process ids for the zebrasrv and daemon processes should be different, but the number of processes is the same as before. 13) sign off, because its less ugly than comment #3 Sponsored-by: Tulong Aklatan Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Works as expected, no regressions found. Pushed to master for 16.11, thanks Mark! Pushed in 16.05. Will be in 16.05.03. Pushed to 3.22.x, will be in 3.22.10 |