Bug 8773 - Add per-instance koha-index-daemon in .deb setup
Summary: Add per-instance koha-index-daemon in .deb setup
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Packaging (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Tomás Cohen Arazi
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on: 8774
Blocks: 11081
  Show dependency treegraph
 
Reported: 2012-09-14 18:21 UTC by Tomás Cohen Arazi
Modified: 2015-06-04 23:32 UTC (History)
9 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (8.92 KB, patch)
2012-09-14 18:37 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (8.88 KB, patch)
2012-09-17 16:11 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (15.82 KB, patch)
2012-09-17 21:11 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (15.82 KB, patch)
2012-09-17 22:13 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
initial test run for koha-common .deb (newer one) (3.27 KB, text/plain)
2012-09-17 23:40 UTC, wajasu
Details
0001-SIGNED-OFF-Bug-8773-Start-per-instance-koha-index-daemon in .deb setup (15.88 KB, patch)
2012-10-03 01:30 UTC, wajasu
Details | Diff | Splinter Review
Bug 8773 - Back to cronjob indexing as default (follow-up) (2.72 KB, patch)
2012-10-23 13:42 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - detect the existence of Koha::Contrib::Tamil (2.34 KB, patch)
2012-10-25 05:50 UTC, Robin Sheat
Details | Diff | Splinter Review
Bug 8773 - Back to cronjob indexing as default (3.24 KB, patch)
2012-10-25 15:07 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - [SIGNED-OFF] detect the existence of Koha::Contrib::Tamil (2.41 KB, patch)
2012-10-29 01:05 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (21.49 KB, patch)
2013-05-20 15:51 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (17.09 KB, patch)
2013-07-19 13:18 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (18.02 KB, patch)
2013-07-19 15:23 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (18.03 KB, patch)
2013-07-19 15:32 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (18.38 KB, patch)
2013-07-19 17:52 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (16.99 KB, patch)
2013-10-01 18:40 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (17.03 KB, patch)
2013-10-20 23:32 UTC, Chris Cormack
Details | Diff | Splinter Review
[PASSED QA] Bug 8773 - Start per-instance koha-index-daemon in .deb setup (17.11 KB, patch)
2014-01-06 09:26 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (16.39 KB, patch)
2014-06-24 16:04 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[SIGNED OFF] Bug 8773 - Start per-instance koha-index-daemon in .deb setup (16.47 KB, patch)
2014-07-08 14:45 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 8773 - Start per-instance koha-index-daemon in .deb setup (16.50 KB, patch)
2014-07-14 01:36 UTC, Chris Cormack
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Tomás Cohen Arazi 2012-09-14 18:21:41 UTC
In order to avoid the use of cronjobs to trigger the incremental indexing of records (does not scale on busy setups, cannot do sub-minute frecaquency indexing), we can use Koha::Contrib::Tamil's koha-index-daemon to do the job.

This should be set automagically by the koha-common package.
Comment 1 Tomás Cohen Arazi 2012-09-14 18:37:00 UTC Comment hidden (obsolete)
Comment 2 Tomás Cohen Arazi 2012-09-17 16:11:10 UTC Comment hidden (obsolete)
Comment 3 Tomás Cohen Arazi 2012-09-17 21:11:25 UTC Comment hidden (obsolete)
Comment 4 Tomás Cohen Arazi 2012-09-17 22:13:22 UTC Comment hidden (obsolete)
Comment 5 Tomás Cohen Arazi 2012-09-17 22:26:41 UTC
Until we get Koha::Contrib::Tamil::IndexerDaemon packed as a .deb file (Bug 8774), it needs to be manually installed for testing this patch.

As building the packages takes some time depending on your hardware, i've uploaded the .deb files to

http://es.koha-community.org/kc_files/

so they can be tested.

You need to grab the koha-common*.deb file, install it (using dpkg -i <file>) and probably create a new koha instance (koha-create --create-db instance). It is worth checking if upon upgrade the running instances get their indexer daemon running automagically.
Comment 6 wajasu 2012-09-17 23:40:05 UTC Comment hidden (obsolete)
Comment 7 Tomás Cohen Arazi 2012-09-17 23:49:35 UTC
(In reply to comment #6)
> Created attachment 12294 [details]
> initial test run for koha-common .deb (newer one)
> 
> I got my VM with the current koha-common. I did dpkg -i <yourkoha-commom> 
> and I've attached what I saw.  
> See initial test run for koha-common .deb (newer one)

Looks like you had a 'biblio' instance previously created, with an uninitialized db. I think that is a bug on the installer / upgrader itself. I'd purge Koha-common, delete /etc/Koha/sites and install the .deb.

other option would be going through the webinstall process prior To the dpkg upgrade.
Comment 8 wajasu 2012-10-03 01:30:26 UTC Comment hidden (obsolete)
Comment 9 Paul Poulain 2012-10-17 16:21:52 UTC
Robin, I'm unable to efficiently QA this patch, and I think other QA team have the same problem. Could you give me your opinion ?

(Note that I could push this patch to 3.10 if it's validated before the end of the week because there's no side-effect risk, so it's not concerned by the current Feature Freeze)
Comment 10 Robin Sheat 2012-10-23 03:10:56 UTC
From reading the patch:
* It updates the man pages. Yay!
* It looks good overall.

My one concern is that it makes the index-daemon the default. As such, I'm not sure if we should be including the patch as is. What I'm thinking I might do (when I've given it a proper test, too) is add a followup script that prevents it being on by default, but makes it very easy to switch.
Comment 11 Robin Sheat 2012-10-23 03:27:00 UTC
I forgot to add that this will allow larger scale testing, and when everything is shown to be happy, we can make it the default. 

Oh, and an issue that I just thought of: on pre-existing installations where the user has modified /etc/cron.d/koha-common and so it doesn't get commented out, we will end up running both the indexing cron job and the daemon. Is this likely to make things asplode?
Comment 12 Robin Sheat 2012-10-23 05:19:09 UTC
More things I've noticed: I'd be quite reluctant to make libkoha-contrib-tamil-perl a dependency on this, for two reasons:

1) It's contrib. Contrib things should not be required dependencies. If we make it a standard thing, then we should pull the required parts into Koha.

2) it puts in a number of unrelated scripts, which are generally useful, but are confusing as they are are in /usr/bin when some of them should be in /usr/sbin, and they don't tend to work in the same way as the packages, which means that if people see they're there, they'll wonder why they don't work like everything else that starts with koha-

Other things:
* This patch doesn't update control.in, so libkoha-contrib-tamil-perl package will never be installed anyway, and things won't work.

I'm not able to build it on master (not your fault, master isn't building right now) so haven't made any progress on proper testing.
Comment 13 Tomás Cohen Arazi 2012-10-23 12:36:41 UTC
Robin, is it possible that we comment out the lines that trigger the daemon (and revert the comment of lines in cron.d/koha-common and push it?

We could advertise it as a experimental feature and provide instructions for enabling it. This would have the advantage of allowing further testing and once we sort those bits out (including the possibility of talking to fredericd about refactoring or reorganizing) it could be set as default in future releases.

I left the control.in file alone as we didn't have those packages avaiable. It could be fixed now with a follow-up.
Comment 14 Tomás Cohen Arazi 2012-10-23 13:42:33 UTC Comment hidden (obsolete)
Comment 15 Robin Sheat 2012-10-25 03:02:13 UTC
Tomás and I talked about this, what we're going to do for now is implement this in a way that it's easy for people to test it. I'd like to see it get in, but we can't make Koha depend on Koha::Contrib::Tamil in its current form. I would like to see the Indexer stuff eventually pulled into Koha itself, but it'll need some refactoring to suit that.

Alternately, we could use another index daemon, and this code will be very easy to change to support it. So we win on all counts.

So what I'm going to do is make a followup to this that checks to see if the required stuff is there, and if it isn't, it'll send people to a wiki page that describes enabling it. This way it's available to advanced users, but not getting in the way on a default install.
Comment 16 Robin Sheat 2012-10-25 05:50:48 UTC Comment hidden (obsolete)
Comment 17 Robin Sheat 2012-10-25 05:51:15 UTC
I've found an error in the indexer, it doesn't seem to retain the PERL5LIB environment when it forks off, and fills the logs with:

Can't locate C4/Context.pm in @INC (@INC contains: /etc/perl /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5 /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10 /usr/local/lib/site_perl .) at /usr/share/perl5/Koha/Contrib/Tamil/RecordReader.pm line 16.
BEGIN failed--compilation aborted at /usr/share/perl5/Koha/Contrib/Tamil/RecordReader.pm line 16.
Compilation failed in require at /usr/share/perl5/Koha/Contrib/Tamil/Indexer.pm line 13.
BEGIN failed--compilation aborted at /usr/share/perl5/Koha/Contrib/Tamil/Indexer.pm line 13.
Compilation failed in require at /usr/share/perl5/Koha/Contrib/Tamil/IndexerDaemon.pm line 13.
BEGIN failed--compilation aborted at /usr/share/perl5/Koha/Contrib/Tamil/IndexerDaemon.pm line 13.
Compilation failed in require at /usr/bin/koha-index-daemon line 17.
BEGIN failed--compilation aborted at /usr/bin/koha-index-daemon line 17.
Comment 18 Frédéric Demians 2012-10-25 05:53:18 UTC
This indexer was provided as a 'contribution', hence the module name,
and so the module is not intended to enter as it is in Koha code.

The indexer depends on Moose. If Moose was a standard Koha dependency, I
could include the indexer in the Koha:: namespace, and put the two
scripts, koha-index and koha-index-daemon, directly in Koha binaries
directory. It's a matter of half an hour's work. And I can do it now.
Without Moose dependency, it will take more time and testing, and I
won't be available to do it before a while.

I don't want to enter into a pro/anti Moose flame war, but can't we
have Moose as an 'optional' dependency, usable now from scripts, not CGI
yet?
Comment 19 Frédéric Demians 2012-10-25 14:10:31 UTC
> I've found an error in the indexer, it doesn't seem to retain the
> PERL5LIB environment when it forks off, and fills the logs with:

"forks off": what does it mean? How can I reproduce that?
Comment 20 Tomás Cohen Arazi 2012-10-25 15:07:33 UTC Comment hidden (obsolete)
Comment 21 Tomás Cohen Arazi 2012-10-25 15:12:51 UTC
(In reply to comment #17)
> I've found an error in the indexer, it doesn't seem to retain the PERL5LIB
> environment when it forks off, and fills the logs with:
> 
> Can't locate C4/Context.pm in @INC (@INC contains: /etc/perl
> /usr/local/lib/perl/5.10.1 /usr/local/share/perl/5.10.1 /usr/lib/perl5
> /usr/share/perl5 /usr/lib/perl/5.10 /usr/share/perl/5.10
> /usr/local/lib/site_perl .) at
> /usr/share/perl5/Koha/Contrib/Tamil/RecordReader.pm line 16.

I cannot reproduce, are u using the Koha::Contrib::Tamil package from squeeze-dev repo?
Comment 22 Robin Sheat 2012-10-26 01:42:29 UTC
Frédéric, it forks off when it becomes a daemon. I'm not sure exactly how it's doing that, but I didn't write it :)

Tomás, according to the docs, it _should_ work. So I'll have a closer look at it when I get the chance. I am using the squeeze-dev one.
Comment 23 Tomás Cohen Arazi 2012-10-29 01:05:34 UTC Comment hidden (obsolete)
Comment 24 Tomás Cohen Arazi 2012-10-29 01:08:25 UTC
Patch labeled "Bug 8773 - Back to cronjob indexing as default" is the one to sign-off. It just comments the lines that launch the indexing daemon from init.d.

Regards
To+
Comment 25 Tomás Cohen Arazi 2012-11-15 00:31:24 UTC
C'mon, sign the missing one, it just reverts the default behaviour.
Comment 26 Tomás Cohen Arazi 2013-03-26 22:07:54 UTC
The only missing signoff is the patch that just disables this, to make it an experimental feature. Its a pity we can't provide a way for people to test using koha-index-daemon with the packages. And its a trivial patch actually.
Comment 27 Katrin Fischer 2013-03-26 22:17:29 UTC
Hi Tomas, is it possible the order of patches is not quite right? Shouldn't be the one missing sign-off be last?
Comment 28 Tomás Cohen Arazi 2013-03-26 22:26:53 UTC
(In reply to comment #27)
> Hi Tomas, is it possible the order of patches is not quite right? Shouldn't
> be the one missing sign-off be last?

I think you're right, they got mixed when I signed and uploaded Robin's patch.
Comment 29 Tomás Cohen Arazi 2013-05-20 15:51:14 UTC Comment hidden (obsolete)
Comment 30 Tomás Cohen Arazi 2013-05-20 15:53:23 UTC
Forgot to mention that lavamind helped me with his ideas and testing.
Comment 31 Tomás Cohen Arazi 2013-07-19 13:18:58 UTC Comment hidden (obsolete)
Comment 32 Tomás Cohen Arazi 2013-07-19 13:24:12 UTC
Now we're going live with 3.12 deb packages at UNC, I found a few glitches on the scripts, fixed them. Also unified the scripts into one with action switches after an IRC poll. Thanks Galen and Mark :-D

If anyone is willing to give this patches a ride, feel free to contact me. Also, I've set a repository for 3.12 with this patches applied for our internal usage, but it is public, and everything is explained here http://blogs.unc.edu.ar/koha/lang/es/2013/07/18/llega-koha-3-12-a-la-unc-nuestro-propio-repositorio-deb-tambienkoha-3-12-arrives-to-unc-our-own-deb-repo-too/

Thanks for any input on this.
Comment 33 Tomás Cohen Arazi 2013-07-19 15:23:07 UTC Comment hidden (obsolete)
Comment 34 Tomás Cohen Arazi 2013-07-19 15:32:23 UTC Comment hidden (obsolete)
Comment 35 Tomás Cohen Arazi 2013-07-19 17:52:59 UTC Comment hidden (obsolete)
Comment 36 I'm just a bot 2013-09-29 05:31:36 UTC
Applying: Bug 8773 - Start per-instance koha-index-daemon in .deb setup
Using index info to reconstruct a base tree...
M	debian/koha-common.init
M	debian/koha-common.install
M	debian/scripts/koha-create
Falling back to patching base and 3-way merge...
Auto-merging debian/scripts/koha-create
Auto-merging debian/koha-common.install
Auto-merging debian/koha-common.init
CONFLICT (content): Merge conflict in debian/koha-common.init
Auto-merging debian/koha-common.default
CONFLICT (add/add): Merge conflict in debian/koha-common.default
Patch failed at 0001 Bug 8773 - Start per-instance koha-index-daemon in .deb setup
The copy of the patch that failed is found in:
   /home/christopher/git/koha/.git/rebase-apply/patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Comment 37 Tomás Cohen Arazi 2013-10-01 18:40:27 UTC Comment hidden (obsolete)
Comment 38 Chris Cormack 2013-10-20 23:32:36 UTC Comment hidden (obsolete)
Comment 39 Martin Renvoize 2013-12-20 16:22:23 UTC
Just a little update on this.. I've built some packaages from this patch, and installed onto a clean VM.

* All seems to work as expected using Tamil indexer
* Patches all pass QA script

The last thing I wanted to check was that this patch will also work with the new daemon mode on rebuild_zebra.pl. I believe it should, but I haven't yet had a moment to get my teeth stuck into that piece.
Comment 40 Martin Renvoize 2013-12-20 16:28:00 UTC
Just a little update on this.. I've built some packages from this patch, and installed onto a clean VM.

* All seems to work as expected using Tamil indexer
* Patches all pass QA script

The last thing I wanted to check was that this patch will also work with the new daemon mode on rebuild_zebra.pl. I believe it should, but I haven't yet had a moment to get my teeth stuck into that piece.
Comment 41 Martin Renvoize 2014-01-06 09:26:58 UTC Comment hidden (obsolete)
Comment 42 Martin Renvoize 2014-01-06 09:31:18 UTC
All works as expected, no regressions created.

I didn't get it fully working with the new daemon mode rebuild_zebra.pl, however I don't think that in itself should fail the patch.  The scaffolding to add support for it is all there, but koha-indexer may need some re-factoring  to support the different switches that rebuild_zebra contains compared to Tamils indexer. (I'd actually rpefer to see rebuild_zebra become close to Tamils indexer personally).
Comment 43 Galen Charlton 2014-03-10 20:41:35 UTC
Comment on attachment 23928 [details] [review]
[PASSED QA] Bug 8773 - Start per-instance koha-index-daemon in .deb setup

Review of attachment 23928 [details] [review]:
-----------------------------------------------------------------

::: debian/koha-common.default
@@ +4,5 @@
> +
> +# If you want to use the koha-index-daemon script that is part of the
> +# Koha::Contrib::Tamil package instead of the cronjob, set USE_INDEXER_DAEMON
> +# to "yes". More information can be found here:
> +#     http://wiki.koha-community.org/wiki/PackagesIndexDaemon

The Tamil indexer is not part of Koha core, and given that bugs 11078 and 6435 have been pushed, the indexer that /is/ is core can now act as a respectable daemon.

Consequently, I'm not inclined to push this as is. Please resubmit a version that uses rebuild_zebra.pl as the default indexer.

Setting to in discussion, but I will need a very good reason to budge on sticking with rebuild_zebra.pl -daemon.
Comment 44 Tomás Cohen Arazi 2014-06-24 16:04:14 UTC Comment hidden (obsolete)
Comment 45 Martin Renvoize 2014-07-08 14:45:53 UTC Comment hidden (obsolete)
Comment 46 Martin Renvoize 2014-07-08 14:46:42 UTC
Comment on attachment 29194 [details] [review]
Bug 8773 - Start per-instance koha-index-daemon in .deb setup

Fantastic work Tomas, so glad to see this working it's way in!
Comment 47 Chris Cormack 2014-07-14 01:36:36 UTC
Created attachment 29669 [details] [review]
Bug 8773 - Start per-instance koha-index-daemon in .deb setup

Short:

Launch an indexing daemon (rebuild_zebra.pl -daemon) process for each
enabled instance. Enabling/disabling the use of the indexer is handled
by global configuration variables in /etc/default/koha-common.

Also provides command line tools to manage the running indexer daemons
for your instances.

Long:

Using an indexing daemon avoids launching a new interpreter each time
the cron triggers the indexing, and also allows sub-minute incremental
reindexing, a requirement from our librarians.[1]

Using the indexer daemon could remain "experimental" until it gets more
testing; so is disabled by default initially. To enable the use of the
indexer the user has to tweak the /etc/default/koha-common config file.
Specifically the USE_INDEXER_DAEMON variable, which is clearly explained
in the file.

Frecquency defaults to 5 sec, and can be changed by tweaking the
/etc/default/koha-common config file too.

This patch uses rebuild_zebra.pl in daemon mode, but it is crafted to
allow changing the indexing daemon and passing specific option switches
it might need.

Regards
To+

[1] This is the .deb version of http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=8519

Sponsored-by: Universidad Nacional de Cordoba
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 48 Tomás Cohen Arazi 2014-07-14 12:21:27 UTC
Patch pushed to master.