In order to avoid the use of cronjobs to trigger the incremental indexing of records (does not scale on busy setups, cannot do sub-minute frecaquency indexing), we can use Koha::Contrib::Tamil's koha-index-daemon to do the job. This should be set automagically by the koha-common package.
Created attachment 12248 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup This bug is the .deb version of 8519 [1]. It is motivated by the fact that using 1-minute (or more) frecquency cronjobs to do the incremental indexing of records does not scale for us, nor is accepted by our librarians. Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute reindexing. Frecquency is hardcoded right now. A follow up could provide a way to set it for each instance. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [2], but could be easily adjusted to use another if we got a new one (SolR?). I havent got the time to setup my infrastructure to generate my own debs, and thus haven't got the oportunity to test this patch. If anyone with the skills helps me on setting that I could test them and fix possible issues. If there's something I could change to make this patches better just let me know. Regards To+ [1] http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [2] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Córdoba
Created attachment 12285 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup This bug is the .deb version of 8519 [1]. It is motivated by the fact that using 1-minute (or more) frecquency cronjobs to do the incremental indexing of records does not scale for us, nor is accepted by our librarians. Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute reindexing. Frecquency is hardcoded right now. A follow up could provide a way to set it for each instance. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [2], but could be easily adjusted to use another if we got a new one (SolR?). I havent got the time to setup my infrastructure to generate my own debs, and thus haven't got the oportunity to test this patch. If anyone with the skills helps me on setting that I could test them and fix possible issues. If there's something I could change to make this patches better just let me know. Regards To+ [1] http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [2] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Córdoba
Created attachment 12292 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup This bug is the .deb version of 8519 [1]. It is motivated by the fact that using 1-minute (or more) frecquency cronjobs to do the incremental indexing of records does not scale for us, nor is accepted by our librarians. Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute reindexing. Frecquency is hardcoded right now. A follow up could provide a way to set it for each instance. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [2], but could be easily adjusted to use another if we got a new one (SolR?). I havent got the time to setup my infrastructure to generate my own debs, and thus haven't got the oportunity to test this patch. If anyone with the skills helps me on setting that I could test them and fix possible issues. If there's something I could change to make this patches better just let me know. Regards To+ [1] http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [2] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Córdoba
Created attachment 12293 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup This bug is the .deb version of 8519 [1]. It is motivated by the fact that using 1-minute (or more) frecquency cronjobs to do the incremental indexing of records does not scale for us, nor is accepted by our librarians. Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute reindexing. Frecquency is hardcoded right now. A follow up could provide a way to set it for each instance. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [2], but could be easily adjusted to use another if we got a new one (SolR?). I havent got the time to setup my infrastructure to generate my own debs, and thus haven't got the oportunity to test this patch. If anyone with the skills helps me on setting that I could test them and fix possible issues. If there's something I could change to make this patches better just let me know. Regards To+ [1] http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [2] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Córdoba
Until we get Koha::Contrib::Tamil::IndexerDaemon packed as a .deb file (Bug 8774), it needs to be manually installed for testing this patch. As building the packages takes some time depending on your hardware, i've uploaded the .deb files to http://es.koha-community.org/kc_files/ so they can be tested. You need to grab the koha-common*.deb file, install it (using dpkg -i <file>) and probably create a new koha instance (koha-create --create-db instance). It is worth checking if upon upgrade the running instances get their indexer daemon running automagically.
Created attachment 12294 [details] initial test run for koha-common .deb (newer one) I got my VM with the current koha-common. I did dpkg -i <yourkoha-commom> and I've attached what I saw. See initial test run for koha-common .deb (newer one)
(In reply to comment #6) > Created attachment 12294 [details] > initial test run for koha-common .deb (newer one) > > I got my VM with the current koha-common. I did dpkg -i <yourkoha-commom> > and I've attached what I saw. > See initial test run for koha-common .deb (newer one) Looks like you had a 'biblio' instance previously created, with an uninitialized db. I think that is a bug on the installer / upgrader itself. I'd purge Koha-common, delete /etc/Koha/sites and install the .deb. other option would be going through the webinstall process prior To the dpkg upgrade.
Created attachment 12654 [details] [review] 0001-SIGNED-OFF-Bug-8773-Start-per-instance-koha-index-daemon in .deb setup I built a fresh debian squeeze, that only had mysql and apache. I downloed the http://es.koha-community.org/kc_files/koha-common_3.9.0-046~git+20120917191354.7807a548_all.deb and scp'ed it to the debian VM in /. I alreadu had the koha-community squeeze main in sources.list As root did: apt-get update apt-get -f install dpkg -i /koha-common_3.9.0-046~git+20120917191354.7807a548_all.deb (I got a lot mof missing dependecies reported and a n error at the end) ethyian noted that dpkg doesn't install those, so i just did: apt-get -f install and answered Yes. and watched all of them get installed, including koha-common which installed without a hitch. zebra and indexer services were started. I created a db instance and the web installer complained about Template::Plugin::HtmlToText not being installed (reqd), but that is recent and should not fail this patch. Possibly improvements to note: The indexer will respawn every 30 seconds generating entries in its log file. Once the index is rebuilt the indexer is happy. But that could be noted with install instructions. Nice job! wajasu
Robin, I'm unable to efficiently QA this patch, and I think other QA team have the same problem. Could you give me your opinion ? (Note that I could push this patch to 3.10 if it's validated before the end of the week because there's no side-effect risk, so it's not concerned by the current Feature Freeze)
From reading the patch: * It updates the man pages. Yay! * It looks good overall. My one concern is that it makes the index-daemon the default. As such, I'm not sure if we should be including the patch as is. What I'm thinking I might do (when I've given it a proper test, too) is add a followup script that prevents it being on by default, but makes it very easy to switch.
I forgot to add that this will allow larger scale testing, and when everything is shown to be happy, we can make it the default. Oh, and an issue that I just thought of: on pre-existing installations where the user has modified /etc/cron.d/koha-common and so it doesn't get commented out, we will end up running both the indexing cron job and the daemon. Is this likely to make things asplode?
More things I've noticed: I'd be quite reluctant to make libkoha-contrib-tamil-perl a dependency on this, for two reasons: 1) It's contrib. Contrib things should not be required dependencies. If we make it a standard thing, then we should pull the required parts into Koha. 2) it puts in a number of unrelated scripts, which are generally useful, but are confusing as they are are in /usr/bin when some of them should be in /usr/sbin, and they don't tend to work in the same way as the packages, which means that if people see they're there, they'll wonder why they don't work like everything else that starts with koha- Other things: * This patch doesn't update control.in, so libkoha-contrib-tamil-perl package will never be installed anyway, and things won't work. I'm not able to build it on master (not your fault, master isn't building right now) so haven't made any progress on proper testing.
Robin, is it possible that we comment out the lines that trigger the daemon (and revert the comment of lines in cron.d/koha-common and push it? We could advertise it as a experimental feature and provide instructions for enabling it. This would have the advantage of allowing further testing and once we sort those bits out (including the possibility of talking to fredericd about refactoring or reorganizing) it could be set as default in future releases. I left the control.in file alone as we didn't have those packages avaiable. It could be fixed now with a follow-up.
Created attachment 13009 [details] [review] Bug 8773 - Back to cronjob indexing as default (follow-up) Following Robin's comment I provide a follow up that restores cronjob as default method for updating indexes. The lines that control the koha-index-daemon are properly left and comented out so the users that want to test this feature have everything already set. Sponsored-by: Universidad Nacional de Córdoba
Tomás and I talked about this, what we're going to do for now is implement this in a way that it's easy for people to test it. I'd like to see it get in, but we can't make Koha depend on Koha::Contrib::Tamil in its current form. I would like to see the Indexer stuff eventually pulled into Koha itself, but it'll need some refactoring to suit that. Alternately, we could use another index daemon, and this code will be very easy to change to support it. So we win on all counts. So what I'm going to do is make a followup to this that checks to see if the required stuff is there, and if it isn't, it'll send people to a wiki page that describes enabling it. This way it's available to advanced users, but not getting in the way on a default install.
Created attachment 13031 [details] [review] Bug 8773 - detect the existence of Koha::Contrib::Tamil If the required module isn't included, then it directs people to have a look at the wiki page that provides instructions on how to configure the index daemon.
I've found an error in the indexer, it doesn't seem to retain the PERL5LIB environment when it forks off, and fills the logs with: Can't locate C4/Context.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at /usr/share/perl5/Koha/Contrib/Tamil/RecordReader.pm line 16. BEGIN failed--compilation aborted at /usr/share/perl5/Koha/Contrib/Tamil/RecordReader.pm line 16. Compilation failed in require at /usr/share/perl5/Koha/Contrib/Tamil/Indexer.pm line 13. BEGIN failed--compilation aborted at /usr/share/perl5/Koha/Contrib/Tamil/Indexer.pm line 13. Compilation failed in require at /usr/share/perl5/Koha/Contrib/Tamil/IndexerDaemon.pm line 13. BEGIN failed--compilation aborted at /usr/share/perl5/Koha/Contrib/Tamil/IndexerDaemon.pm line 13. Compilation failed in require at /usr/bin/koha-index-daemon line 17. BEGIN failed--compilation aborted at /usr/bin/koha-index-daemon line 17.
This indexer was provided as a 'contribution', hence the module name, and so the module is not intended to enter as it is in Koha code. The indexer depends on Moose. If Moose was a standard Koha dependency, I could include the indexer in the Koha:: namespace, and put the two scripts, koha-index and koha-index-daemon, directly in Koha binaries directory. It's a matter of half an hour's work. And I can do it now. Without Moose dependency, it will take more time and testing, and I won't be available to do it before a while. I don't want to enter into a pro/anti Moose flame war, but can't we have Moose as an 'optional' dependency, usable now from scripts, not CGI yet?
> I've found an error in the indexer, it doesn't seem to retain the > PERL5LIB environment when it forks off, and fills the logs with: "forks off": what does it mean? How can I reproduce that?
Created attachment 13050 [details] [review] Bug 8773 - Back to cronjob indexing as default Following Robin's comment I provide a follow up that restores cronjob as default method for updating indexes. The lines that control the koha-index-daemon are properly left and comented out so the users that want to test this feature have everything already set. Sponsored-by: Universidad Nacional de Córdoba
(In reply to comment #17) > I've found an error in the indexer, it doesn't seem to retain the PERL5LIB > environment when it forks off, and fills the logs with: > > Can't locate C4/Context.pm in @INC (@INC contains: /etc/perl > /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 > /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 > /usr/local/lib/site_perl .) at > /usr/share/perl5/Koha/Contrib/Tamil/RecordReader.pm line 16. I cannot reproduce, are u using the Koha::Contrib::Tamil package from squeeze-dev repo?
Frédéric, it forks off when it becomes a daemon. I'm not sure exactly how it's doing that, but I didn't write it :) Tomás, according to the docs, it _should_ work. So I'll have a closer look at it when I get the chance. I am using the squeeze-dev one.
Created attachment 13089 [details] [review] Bug 8773 - [SIGNED-OFF] detect the existence of Koha::Contrib::Tamil If the required module isn't included, then it directs people to have a look at the wiki page that provides instructions on how to configure the index daemon. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Patch labeled "Bug 8773 - Back to cronjob indexing as default" is the one to sign-off. It just comments the lines that launch the indexing daemon from init.d. Regards To+
C'mon, sign the missing one, it just reverts the default behaviour.
The only missing signoff is the patch that just disables this, to make it an experimental feature. Its a pity we can't provide a way for people to test using koha-index-daemon with the packages. And its a trivial patch actually.
Hi Tomas, is it possible the order of patches is not quite right? Shouldn't be the one missing sign-off be last?
(In reply to comment #27) > Hi Tomas, is it possible the order of patches is not quite right? Shouldn't > be the one missing sign-off be last? I think you're right, they got mixed when I signed and uploaded Robin's patch.
Created attachment 18258 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[2] Using the indexer daemon should remain "experimental" so is disabled by default. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file (introduced by this patch). Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Update frecquency defaults to 30 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [3], but could be easily adjusted to use another if we got a new one (SolR?). As Robin proposed it check for availability of the relevant files before trying to spawn the daemon. If there's something I could change to make this patches better just let me know. Note: there was a problem Robin found regarding the spawned processes forgetting the PERL5LIB and/or KOHA_CONF env variables, this is fixed in this version with the inclusion of the 'export' keyword for the relevant variables. Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Córdoba
Forgot to mention that lavamind helped me with his ideas and testing.
Created attachment 19794 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Also provide command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[2] Using the indexer daemon should remain "experimental" so is disabled by default. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file (introduced by this patch). Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Update frecquency defaults to 30 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [3], but could be easily adjusted to use another if we got a new one (SolR?). As Robin proposed it checks for availability of the relevant files before trying to spawn the daemon. If there's something I could change to make this patches better just let me know. Note: there was a problem Robin found regarding the spawned processes forgetting the PERL5LIB and/or KOHA_CONF env variables, this is fixed in this version with the inclusion of the 'export' keyword for the relevant variables. Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Cordoba
Now we're going live with 3.12 deb packages at UNC, I found a few glitches on the scripts, fixed them. Also unified the scripts into one with action switches after an IRC poll. Thanks Galen and Mark :-D If anyone is willing to give this patches a ride, feel free to contact me. Also, I've set a repository for 3.12 with this patches applied for our internal usage, but it is public, and everything is explained here http://blogs.unc.edu.ar/koha/lang/es/2013/07/18/llega-koha-3-12-a-la-unc-nuestro-propio-repositorio-deb-tambienkoha-3-12-arrives-to-unc-our-own-deb-repo-too/ Thanks for any input on this.
Created attachment 19798 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables. Also provide command line tools to manage the running indexer daemons for your instances. Edit: fixed koha-create to call the koha-indexer script instead, and added -q to make koha-inxeder quiet (useful for the init script). Regards To+
Created attachment 19800 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Also provide command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[2] Using the indexer daemon should remain "experimental" so is disabled by default. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file (introduced by this patch). Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Update frecquency defaults to 30 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [3], but could be easily adjusted to use another if we got a new one (SolR?). As Robin proposed it checks for availability of the relevant files before trying to spawn the daemon. If there's something I could change to make this patches better just let me know. Note: there was a problem Robin found regarding the spawned processes forgetting the PERL5LIB and/or KOHA_CONF env variables, this is fixed in this version with the inclusion of the 'export' keyword for the relevant variables. Edit: fixed koha-create to call the koha-indexer script instead, and added -q to make koha-inxeder quiet (useful for the init script). Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Cordoba
Created attachment 19805 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Also provide command line tools to manage the running indexer daemons for your instances. Edit: fixed koha-create to call the koha-indexer script instead, and added -q to make koha-inxeder quiet (useful for the init script). Added references to /etc/default/koha-common at koha-create to test for the indexer daemon configuration. Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Cordoba
Applying: Bug 8773 - Start per-instance koha-index-daemon in .deb setup Using index info to reconstruct a base tree... M debian/koha-common.init M debian/koha-common.install M debian/scripts/koha-create Falling back to patching base and 3-way merge... Auto-merging debian/scripts/koha-create Auto-merging debian/koha-common.install Auto-merging debian/koha-common.init CONFLICT (content): Merge conflict in debian/koha-common.init Auto-merging debian/koha-common.default CONFLICT (add/add): Merge conflict in debian/koha-common.default Patch failed at 0001 Bug 8773 - Start per-instance koha-index-daemon in .deb setup The copy of the patch that failed is found in: /home/christopher/git/koha/.git/rebase-apply/patch When you have resolved this problem, run "git am --continue". If you prefer to skip this patch, run "git am --skip" instead. To restore the original branch and stop patching, run "git am --abort".
Created attachment 21704 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Also provide command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[2] Using the indexer daemon should remain "experimental" so is disabled by default. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file (introduced by this patch). Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Update frecquency defaults to 30 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [3], but could be easily adjusted to use another if we got a new one (SolR?). As Robin proposed it checks for availability of the relevant files before trying to spawn the daemon. If there's something I could change to make this patches better just let me know. Note: there was a problem Robin found regarding the spawned processes forgetting the PERL5LIB and/or KOHA_CONF env variables, this is fixed in this version with the inclusion of the 'export' keyword for the relevant variables. Edit: fixed koha-create to call the koha-indexer script instead, and added -q to make koha-inxeder quiet (useful for the init script). Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Cordoba
Created attachment 22135 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Also provide command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[2] Using the indexer daemon should remain "experimental" so is disabled by default. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file (introduced by this patch). Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Update frecquency defaults to 30 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [3], but could be easily adjusted to use another if we got a new one (SolR?). As Robin proposed it checks for availability of the relevant files before trying to spawn the daemon. If there's something I could change to make this patches better just let me know. Note: there was a problem Robin found regarding the spawned processes forgetting the PERL5LIB and/or KOHA_CONF env variables, this is fixed in this version with the inclusion of the 'export' keyword for the relevant variables. Edit: fixed koha-create to call the koha-indexer script instead, and added -q to make koha-inxeder quiet (useful for the init script). Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>
Just a little update on this.. I've built some packaages from this patch, and installed onto a clean VM. * All seems to work as expected using Tamil indexer * Patches all pass QA script The last thing I wanted to check was that this patch will also work with the new daemon mode on rebuild_zebra.pl. I believe it should, but I haven't yet had a moment to get my teeth stuck into that piece.
Just a little update on this.. I've built some packages from this patch, and installed onto a clean VM. * All seems to work as expected using Tamil indexer * Patches all pass QA script The last thing I wanted to check was that this patch will also work with the new daemon mode on rebuild_zebra.pl. I believe it should, but I haven't yet had a moment to get my teeth stuck into that piece.
Created attachment 23928 [details] [review] [PASSED QA] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch a koha-index-daemon (from Koha::Contrib::tamil) process for each enabled instance. Enabling/disabling the use of the indexer is handled using a new(ly introduced) /etc/default/koha-common config file that can be used for other stuff too, and is the Debian way of handling init scripts control variables[1]. Also provide command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[2] Using the indexer daemon should remain "experimental" so is disabled by default. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file (introduced by this patch). Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Update frecquency defaults to 30 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch relies on Koha::Contrib::Tamil's koha-index-daemon script [3], but could be easily adjusted to use another if we got a new one (SolR?). As Robin proposed it checks for availability of the relevant files before trying to spawn the daemon. If there's something I could change to make this patches better just let me know. Note: there was a problem Robin found regarding the spawned processes forgetting the PERL5LIB and/or KOHA_CONF env variables, this is fixed in this version with the inclusion of the 'export' keyword for the relevant variables. Edit: fixed koha-create to call the koha-indexer script instead, and added -q to make koha-inxeder quiet (useful for the init script). Regards To+ [1] Section 9.3.2 of http://www.debian.org/doc/debian-policy/ch-opersys.html [2] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 [3] The default cronjob that is set using the debian/koha-common.cron.d file is being disbaled by this patch to avoid collision. Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
All works as expected, no regressions created. I didn't get it fully working with the new daemon mode rebuild_zebra.pl, however I don't think that in itself should fail the patch. The scaffolding to add support for it is all there, but koha-indexer may need some re-factoring to support the different switches that rebuild_zebra contains compared to Tamils indexer. (I'd actually rpefer to see rebuild_zebra become close to Tamils indexer personally).
Comment on attachment 23928 [details] [review] [PASSED QA] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Review of attachment 23928 [details] [review]: ----------------------------------------------------------------- ::: debian/koha-common.default @@ +4,5 @@ > + > +# If you want to use the koha-index-daemon script that is part of the > +# Koha::Contrib::Tamil package instead of the cronjob, set USE_INDEXER_DAEMON > +# to "yes". More information can be found here: > +# http://wiki.koha-community.org/wiki/PackagesIndexDaemon The Tamil indexer is not part of Koha core, and given that bugs 11078 and 6435 have been pushed, the indexer that /is/ is core can now act as a respectable daemon. Consequently, I'm not inclined to push this as is. Please resubmit a version that uses rebuild_zebra.pl as the default indexer. Setting to in discussion, but I will need a very good reason to budge on sticking with rebuild_zebra.pl -daemon.
Created attachment 29194 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch an indexing daemon (rebuild_zebra.pl -daemon) process for each enabled instance. Enabling/disabling the use of the indexer is handled by global configuration variables in /etc/default/koha-common. Also provides command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[1] Using the indexer daemon could remain "experimental" until it gets more testing; so is disabled by default initially. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file. Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Frecquency defaults to 5 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch uses rebuild_zebra.pl in daemon mode, but it is crafted to allow changing the indexing daemon and passing specific option switches it might need. Regards To+ [1] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 Sponsored-by: Universidad Nacional de Cordoba
Created attachment 29575 [details] [review] [SIGNED OFF] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch an indexing daemon (rebuild_zebra.pl -daemon) process for each enabled instance. Enabling/disabling the use of the indexer is handled by global configuration variables in /etc/default/koha-common. Also provides command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[1] Using the indexer daemon could remain "experimental" until it gets more testing; so is disabled by default initially. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file. Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Frecquency defaults to 5 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch uses rebuild_zebra.pl in daemon mode, but it is crafted to allow changing the indexing daemon and passing specific option switches it might need. Regards To+ [1] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment on attachment 29194 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Fantastic work Tomas, so glad to see this working it's way in!
Created attachment 29669 [details] [review] Bug 8773 - Start per-instance koha-index-daemon in .deb setup Short: Launch an indexing daemon (rebuild_zebra.pl -daemon) process for each enabled instance. Enabling/disabling the use of the indexer is handled by global configuration variables in /etc/default/koha-common. Also provides command line tools to manage the running indexer daemons for your instances. Long: Using an indexing daemon avoids launching a new interpreter each time the cron triggers the indexing, and also allows sub-minute incremental reindexing, a requirement from our librarians.[1] Using the indexer daemon could remain "experimental" until it gets more testing; so is disabled by default initially. To enable the use of the indexer the user has to tweak the /etc/default/koha-common config file. Specifically the USE_INDEXER_DAEMON variable, which is clearly explained in the file. Frecquency defaults to 5 sec, and can be changed by tweaking the /etc/default/koha-common config file too. This patch uses rebuild_zebra.pl in daemon mode, but it is crafted to allow changing the indexing daemon and passing specific option switches it might need. Regards To+ [1] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519 Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Patch pushed to master.