Bug 27140 - koha-zebra doesn't properly stop Zebra
Summary: koha-zebra doesn't properly stop Zebra
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-12-03 04:33 UTC by David Cook
Modified: 2023-07-01 14:28 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2020-12-03 04:33:38 UTC
So I just tried "koha-zebra --restart kohadev" after making a change to log levels to include request in koha-conf.xml in koha-testing-docker and it's a mess:

root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+   808     1  0 02:04 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+   840   808  0 02:04 ?        00:00:00 [zebrasrv] <defunct>
kohadev+  5183     1  0 04:28 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root      5373  3959  0 04:31 pts/1    00:00:00 grep zebrasrv

Using "kill -9", I kill off all those.

I run "koha-zebra --start kohadev":

root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+  5418     1  0 04:31 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+  5422  5418  0 04:31 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root      5424  3959  0 04:31 pts/1    00:00:00 grep zebrasrv

That's much better now.
Comment 1 Marcel de Rooy 2020-12-03 06:37:11 UTC
Remember some similar experience. But is the info clear enough for someone to solve this ticket ? :)
Comment 2 David Cook 2020-12-04 02:49:26 UTC
(In reply to Marcel de Rooy from comment #1)
> Remember some similar experience. But is the info clear enough for someone
> to solve this ticket ? :)

Good point heh.

Interestingly... I just tried "koha-zebra --stop -v kohadev" and "koha-zebra --start -v kohadev" and it worked correctly.

"koha-zebra --restart -v kohadev" just worked too...

--

Aha... I just went to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test and then I tried "koha-zebra --stop -v kohadev" but it didn't work.

Evidence:
------------------------------ ---------------------------------------- -------------------------------------
root@kohadevbox:koha(bug_12430)$ koha-zebra --start -v kohadev
[ ok ] Starting Koha Zebra daemon for kohadev:.
root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+ 25286     1  1 02:38 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25308 25286  0 02:38 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25310  3959  0 02:38 pts/1    00:00:00 grep zebrasrv
------------------------------ ---------------------------------------- -------------------------------------
root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+ 25286     1  0 02:38 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25308 25286  0 02:38 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25315 25308  0 02:38 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25320  3959  0 02:38 pts/1    00:00:00 grep zebrasrv
------------------------------ ---------------------------------------- -------------------------------------
root@kohadevbox:koha(bug_12430)$ koha-zebra --stop -v kohadev
[ ok ] Stopping Koha Zebra daemon for kohadev:.
root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+ 25286     1  0 02:38 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25308 25286  0 02:38 ?        00:00:00 [zebrasrv] <defunct>
kohadev+ 25315     1  0 02:38 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25357  3959  0 02:38 pts/1    00:00:00 grep zebrasrv
------------------------------ ---------------------------------------- -------------------------------------

It looks like the daemon stop killed the original zebrasrv process but it didn't kill the child process of the original zebrasrv process...

If I kill the child process (pid 25315), then everything will terminate:

root@kohadevbox:koha(bug_12430)$ kill -9 25315
root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
root     25417  3959  0 02:42 pts/1    00:00:00 grep zebrasrv

------------------------------ ---------------------------------------- -------------------------------------
Comment 3 David Cook 2020-12-04 02:53:02 UTC
If you manually try to kill the parent zebrasrv, you get the same problem.

------------------------------ ---------------------------------------- ------------------
root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+ 25532     1  0 02:48 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25553 25532  0 02:48 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25560 25553  1 02:48 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25563  3959  0 02:48 pts/1    00:00:00 grep zebrasrv
------------------------------ ---------------------------------------- ------------------
root@kohadevbox:koha(bug_12430)$ kill 25553
------------------------------ ---------------------------------------- ------------------
root@kohadevbox:koha(bug_12430)$ ps -efww | grep "zebrasrv"
kohadev+ 25532     1  0 02:48 ?        00:00:00 daemon --name=kohadev-koha-zebra --pidfiles=/var/run/koha/kohadev/ --errlog=/var/log/koha/kohadev/zebra-error.log --output=/var/log/koha/kohadev/zebra-output.log --verbose=1 --respawn --del
ay=30 --user=kohadev-koha.kohadev-koha -- /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
kohadev+ 25553 25532  0 02:48 ?        00:00:00 [zebrasrv] <defunct>
kohadev+ 25560     1  0 02:48 ?        00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25575  3959  0 02:49 pts/1    00:00:00 grep zebrasrv
------------------------------ ---------------------------------------- ------------------

If we manually kill the parent zebrasrv process, it becomes a zombie and pid 25560 is adopted by pid 1.
Comment 4 David Cook 2020-12-04 02:57:10 UTC
If I manually run zebrasrv by hand I'll get the same result which shows that zebrasrv isn't set up to propagate SIGTERM to its child processes. 

/usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml

Visit http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=test

root@kohadevbox:kohadevbox$ ps -efww | grep "zebrasrv"
root     25693  3959  0 02:52 pts/1    00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25697 25693  0 02:53 pts/1    00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25712 25617  0 02:53 pts/2    00:00:00 grep zebrasrv
root@kohadevbox:kohadevbox$ kill 25693
root@kohadevbox:kohadevbox$ ps -efww | grep "zebrasrv"
root     25697     1  0 02:53 pts/1    00:00:00 /usr/bin/zebrasrv -v none,fatal,warn,request -k 1024 -f /etc/koha/sites/kohadev/koha-conf.xml
root     25727 25617  0 02:53 pts/2    00:00:00 grep zebrasrv

------------------------------ ---------------------------------------- ------------------

On koha-testing-docker, we are using 2.0.59 (and I have mine set up for ICU atm):

root@kohadevbox:kohadevbox$ zebraidx -V
Zebra 2.0.59
(C) 1994-2014, Index Data
Zebra is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
SHA1 ID: c00bfddbf0f3608340d61298acc61dafb167f9b2
Using ICU

Looking at the Zebra news https://github.com/indexdata/idzebra/blob/master/NEWS, I don't see anything about SIGTERM specifically.
Comment 5 David Cook 2020-12-04 03:14:36 UTC
Ultimately, I'd say that we can't reliably use the "daemon" utility to stop "zebrasrv", since "daemon" just sends a SIGTERM signal to "zebrasrv" and "zebrasrv" does not propagate SIGTERM to its child processes.

This prevents "zebrasrv" from being stopped by "daemon --stop".

Since "zebrasrv" comes with its own -l logfile, -D daemon, and -p pidfile options, I think it would be trivial to just manage zebrasrv via a koha-zebra script that does not use the "daemon" utility.

Of course, we'd have to SIGKILL zebrasrv to get it to really stop. 

Interestingly, looking at https://github.com/indexdata/idzebra/blob/master/etc/init.d/idzebra-init-script shows that they try to "stop" their zebrasrv by just using SIGTERM. (It's also a poorly written service overall...)

My guess is that they think it works correctly but I bet that it doesn't... 

We're running Zebra 2.2.2 on Ubuntu and it looks like someone has provided a different script for /etc/init.d/zebrasrv which uses "start-stop-daemon".

This is how that service stops zebrasrv:

start-stop-daemon --stop --quiet --retry=TERM/30/KILL/5 --pidfile $PIDFILE --name $NAME

However, SIGKILL probably won't work any better as it won't propagate to the child process.