Description
Tomás Cohen Arazi (tcohen)
2012-07-30 12:36:04 UTC
Created attachment 11195 [details] [review] Bug 8519 - Make koha-index-daemon run on startup Remove obsolete zebraqueue_daemon.pl obsolete script too. Tested to work on an up-to-date Ubuntu 12.04. Created attachment 11197 [details] [review] Bug 8519 - Make koha-index-daemon run on startup Remove obsolete zebraqueue_daemon.pl obsolete script too. Tested to work on an up-to-date Ubuntu 12.04. To test this patch, install Koha, create a symlink for this script: On Ubuntu: $ ln -s /usr/share/koha/bin/koha-zebra-indexdaemon-ctl.sh /etc/init.d/koha-zebra-indexdaemon $ update-rc.d koha-zebra-indexdaemon defaults $ service koha-zebra-indexdaemon start Then: - Check it started without errors in the logs (/var/log/koha/koha-indexdaemon*) - Do some cataloguing (biblios, authorities) and see if they are properly indexed by zebra. I attempted to test, but it had a few of problems, with my "standard" fresh install. 1) make install failed, complaining that: "Can't open blib/SCRIPT_DIR/koha-zebraqueue-ctl.sh for read at rewrite-config.PL line 184". It seems the Makefile.PL processes it at line 525, but that was removed in the patch. It probably needs to be changed refer to the koha-zebra-indexdaemon-ctl.sh to replace the rewrite_config fields. 2) I changed it in my test environment, and "make install" ran, and I symlinked/ran the script but got : 20120810 15:39:25 koha-zebra-indexdaemon-ctl-koha: client (pid 11935) exited with 1 status ... 20120810 15:39:25 koha-zebra-indexdaemon-ctl-koha: client (pid 11940) exited with 1 status 20120810 15:39:25 koha-zebra-indexdaemon-ctl-koha: terminating too quickly, waiting 30 seconds koha-index-daemon is not anywhere to be found in the patch. so 3) jcamins on IRC suggested that Koha:Contrib:Tamil contained the missing script, so I used cpan in my test vm. That included a lot of stuff. Maybe we only require the Koha:Contrib:Tamil:Indexer and Koha:Contrib:Tamil:IndexerDaemon components. After that I got the koha-zebra-indexdaemon script running. But, it was failing which is the default directory. So I added to misc/bin/koha-zebra-indexdaemon-ctl.sh as part of the INDEXDAEMONOPTS --directory /var/tmp/$DBNAME-zebraqueue-index which allowed the tamil contribs indexdaemon to create it. I checked the zebraqueue tables and after the 30 second timeout, it processed another z3950 cataloging import. Great! 4) Other things to notice: find kohaclonedir -name "*.*" -exec grep -Hn "zebraqueue" {} \; to find other places that mentioned zebra queue related things, and saw a ./misc/cronjobs/zebraqueue_start.pl This is probably obsolete as well, and should be removed. The INSTALL texts need to be updated, and one refers you to the wiki for configuration. So that need to be addressed. Possibly as a followup patch. I will let the daemon run over the weekend to see if it behaves. But maybe we want someone to write a load "test". I'm going to bring things up next week to see if anything I mentioned needs to be addressed from the patch writer and community. Created attachment 11551 [details] [review] Bug 8519 - Make koha-index-daemon run on startup Added a new koha-index-daemon-ctl.sh script that uses the 'koha-index-daemon' provided by Koha::Contrib::Tamil (install via CPAN) to process the indexing queue ('zebraqueue' for now, 'indexqueue' in the future I guess :-P). This script could be easily modified in the future for using another index queue processing script (a more abstract indexing script for both Solr and Zebra, etc). I removed all zebra-ish stuff from it. It removes the obsolete zebraqueue_daemon.pl and koha-zebraqueue-ctl.sh obsolete scripts too. Several files are modified to address te removal/addition of these files. I didn't run the install procedure as I was working on my laptop with a dev setup, just set the symlinks. Now fixed things as proposed by wajasu on comment #4. Any other suggestion please let me know. Documentation patches will be provided as followups, once Mark Tompsett's docs are pushed to avoid continous rebasing. Tested to work on an up-to-date Ubuntu 12.04. Sponsored-by: Universidad Nacional de Córdoba Created attachment 11558 [details] [review] SIGNED-OFF-Bug-8519-Make-koha-index-daemon-run-on-st Applied this latest patch from tcohen and it was functional. 1) downloaded latest Bug-8519---Make-koha-index-daemon-run-on-startup.patch 2) git clone git://git.koha-community.org/koha.git kohaclone on Aug 14th 2012 3) cd kohaclone 4) git am -i -u -3 Bug-8519---Make-koha-index-daemon-run-on-startup.patch 5) perl Makefile.PL 6) make (I didn't perform make test, as it failed without memcached installed) 7) make install 8) ln -s /usr/share/koha/bin/koha-index-daemon-ctl.sh /etc/init.d/koha-index-daemon 9) With my zebrasrv already running... 10) /etc/init.d/koha-index-daemon start 11) Then watched /var/log/koha/koha-index-daemon-output.log Note: periodically as configured it will try to create to index, creating the --directory /var/tmp was used as its standard with the least room for permissions error as we've had in the past. 12) performed cataloging operation and imported biblio from z3950 server 13) mysql -uroot -p koha > select * from zebraqueue; showed my reently cataloged record done is 0 14) did a catalog search for my record (not found) 13) after 30 seconds > select * from zebraqueue; showed my reently cataloged record done is 1 15) did a catalog search for my record (FOUND!!!) Yes!!! 16) I did a few more cataloging ops and found them all. No erros in Tamil indexer, daemon keeps running. 17) Ran it overnight, with no problems. Notes: I had already done a cpan install of Koha:Contrib:Tamil version 0.024 PERL5LIB=/usr/share/koha/lib in my standard install, needs to be defined during the Koha:Contrib:Tamil install. As folks on IRC suggested, we need a followup patch for this, in PerlModules.pm since its optional. Also, I am not sure if the perl module dependency should be on a subset of Koha:Contrib:Tamil, such as Indexer and IndexerDaemon (see cpan) koha-common package has its own daemon setup, which supports multiple koha installs on the same box. We might reference that in a future design. tcohen added a --directory option in the daemon script by my request to get things operational. If there is a preference to have configuration centrally managed in KOHA_CONF, we might try using xmlstarlet(new koha dependency?) to query a new KOHA_CONF section <config><indexerdir>/var/tmp/koha</indexerdir></config> OR allow rewrite_config.pl to fill it in. We may want to create a followup patch to remove misc/cronjobs/zebraqueue_start.pl if its a variation of rebuild_zebra.pl Observations: It seems the indexer cleans up records exported(changes/deleted records) from the database, so no long term space issues. The logs only really get large if an error occurs when misconfigured (no directory can be created, or zebrasrv is down?) Greate patch tcohen!!! Now we have more stable and more frequent index updates. wajasu Created attachment 11560 [details] [review] Bug 8519 - Remove zebraqueue_start.pl script (cleanup followup, not important for the bug) Asked by wajasu, remove remaining obsolete zebraqueue stuff. About the koha-common package, I think I'm not sure what actually is inside of it :-D Last time I saw it, I found it setting a cronjob that looped around enabled instances and run rebuild_zebra against them. Sponsored-by: Universidad Nacional de Córdoba Yes it runs a cron job that runs rebuild zebra for each instance. So this shouldn't effect it at all its only removing the old deprecated daemon and adding a new optional daemon. The cron job method shouldn't be touched at all, ideally an option to run the daemon with the packages would be good too. But it's mire important to not break existing functionality. Created attachment 11724 [details] [review] 0001-SIGNED-OFF-Bug-8519-Remove-zebraqueue_start.pl-scrip This followup patch removes what is thought to be an obsolete zebrraqueue_start.pl, since /etc/init.d/koha-index-daemon links to a new indexer control script. If we leave this around, it will confuse deployers. I've been running the koha-index-daemon for a week or so. Its working fine. Someone else who is knowledgable might confirm that this: 0001-SIGNED-OFF-Bug-8519-Remove-zebraqueue_start.pl-scrip is the right idea. wajasu Observation: after the "Make-koha-index-daemon-run" and "Remove-zerbaqueue_start" were applied, I still found reference to zebraqueue_start. From the kohaclone git directory: find . -name "*.*" -exec grep -Hn "zebraqueue_start" {} \; results in: ./C4/Biblio.pm:2660:# the table is emptied by misc/cronjobs/zebraqueue_start.pl script ./C4/Biblio.pm:2714: # the table is emptied by misc/cronjobs/zebraqueue_start.pl script ./C4/AuthoritiesMarc.pm:835: # indexing processes like zebraqueue_start.pl so this must be removed/changed to prevent from confusing future development. wajasu I'd like to tell that this patch would also be usefull in the case of using another daemon script (the one for SolR for example). This is only a init script. ive added a new bug '8774' as a blocker for this I wonder what with happen with this bug now we have packages for Fredericd's daemon Created attachment 13156 [details] [review] Bug 8519 - Make koha-index-daemon run on startup [This patch was split out from tcohen's excellent patches for bug 8519 --jcamins 2012/10/31] Added a new koha-index-daemon-ctl.sh script that uses the 'koha-index-daemon' provided by Koha::Contrib::Tamil (install via CPAN) to process the indexing queue ('zebraqueue' for now, 'indexqueue' in the future I guess :-P). This script could be easily modified in the future for using another index queue processing script (a more abstract indexing script for both Solr and Zebra, etc). I removed all zebra-ish stuff from it. Documentation patches will be provided as followups, once Mark Tompsett's docs are pushed to avoid continous rebasing. Tested to work on an up-to-date Ubuntu 12.04. Sponsored-by: Universidad Nacional de Córdoba Signed-off-by: wajasu <matted-34813@mypacks.net> Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Dry QA comment: Cannot test this patch, since it is dependent on distro and koha-index-daemon. (This will be the case for more members of QA team probably.) Things as /lib/lsb/init-functions and daemon are not always present. I tested make test and make install. Make test did not finish properly, but that seems not to be related to this patch. Make install installed the new script in the kohadev bin directory. As far as I know, the t/db_dependent/lib/KohaTest.pm is a deprecated test unit. But I do not see any harm in pushing this patch. It will not break anything if you do not use it. It needs more documentation but that was mentioned already. Setting to Passed QA for consideration by RM. This patch has been pushed to master. This patch shipped with 3.12.0. |