Bug 16814 - Indexer daemon stresses out server with many instances
Summary: Indexer daemon stresses out server with many instances
Status: CLOSED WORKSFORME
Alias: None
Product: Koha
Classification: Unclassified
Component: Packaging (show other bugs)
Version: 16.05
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-06-27 12:38 UTC by Magnus Enger
Modified: 2017-06-14 22:10 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2016-06-27 12:38:21 UTC
Bug 16190 enables the indexer daemon by default for package installations. 

I just upgraded my test server to 16.05.01. This server has 57 instances on it, so it might be a bit of an outlier. But right after the upgrade started the server maxed out on load, memory usage and swap. To the point where everyting became unbearably slow. 

I tried setting INDEXER_TIMEOUT=30 in /etc/default/koha-common and rebooting, but the same problem appeared almost at once. 

Setting USE_INDEXER_DAEMON="no" and rebooting made the server usable again. 

Not sure if this should be considered a problem, or if the problem is the high number of instances on one server. The server is a "Linode 4096" and it has been running this number of instances (which have very low usage) without any problems for a long time prior to the upgrade.
Comment 1 Tomás Cohen Arazi 2016-06-27 12:43:47 UTC
The indexer timeout wouldn't have any impact (the CPU footprint is small for indexer daemons), problem is memory.

I'd say such a highly demanded server is a special situation that the sysadmin should be able to handle. But I'd say you will be trading memory for high CPU usage if you stick to the usual cronjobs on that many instances.

For me this is a WORKSFORME
Comment 2 Mirko Tietgen 2016-06-27 14:22:59 UTC
I have ~25 test instances on a small server with 4 GB ram. I upgraded to 16.05.01, so far I can't see anything like that. 3 CPU cores at almost 0, about 2.4 GB RAM used, no swap.
Comment 3 Tomás Cohen Arazi 2016-06-27 14:25:12 UTC
(In reply to Mirko Tietgen from comment #2)
> I have ~25 test instances on a small server with 4 GB ram. I upgraded to
> 16.05.01, so far I can't see anything like that. 3 CPU cores at almost 0,
> about 2.4 GB RAM used, no swap.

What I mean, is that someone maintaining a 10+ instances server should be able to make de decision to change the enabled-by-default indexer if it was a problem on its deployment infrastructure.
But the feature should definitely remain enabled by default for the average user.
Comment 4 Mirko Tietgen 2016-06-27 14:29:16 UTC
Agreed.
Comment 5 Mirko Tietgen 2016-06-27 14:36:08 UTC
Magnus: do you have a custom cronjob for koha-rebuild-zebra that may be running while you also use the inder daemon? I just found that here and if I had bigger data sets, that would probably make the server sad.
Comment 6 Magnus Enger 2016-06-28 08:07:50 UTC
(In reply to Mirko Tietgen from comment #5)
> Magnus: do you have a custom cronjob for koha-rebuild-zebra that may be
> running while you also use the inder daemon? I just found that here and if I
> had bigger data sets, that would probably make the server sad.

Not custom, but the standard cronjob set up by the packages in /etc/cron.d/koha-common was enabled: 

*/5 * * * * root test -x /usr/sbin/koha-rebuild-zebra && koha-rebuild-zebra -q $(koha-list --enabled)

I just tried commenting out this, setting USE_INDEXER_DAEMON="yes" and rebooting, and load, memory usage and swap went through the roof pretty instantly after the reboot. 

I am fine with WORKSFORME - as I said, a server with this many instances is probably an edge case.

But, should we at least remove or comment out this:

*/5 * * * * root test -x /usr/sbin/koha-rebuild-zebra && koha-rebuild-zebra -q $(koha-list --enabled) 

from /etc/cron.d/koha-common (debian/koha-common.cron.d in git) now that the daemon is activated as default?
Comment 7 Mason James 2016-06-28 08:31:11 UTC
(In reply to Magnus Enger from comment #6)

> But, should we at least remove or comment out this:
> 
> */5 * * * * root test -x /usr/sbin/koha-rebuild-zebra && koha-rebuild-zebra
> -q $(koha-list --enabled) 
> 
> from /etc/cron.d/koha-common (debian/koha-common.cron.d in git) now that the
> daemon is activated as default?

i agree Magnus... i got caught out by this just last week! :0)
Comment 8 Magnus Enger 2016-06-28 08:37:16 UTC
I can do separate bug and patch for that.
Comment 9 Magnus Enger 2016-06-28 08:41:20 UTC
Bug 16823 - Comment out koha-rebuild-zebra in debian/koha-common.cron.d
Comment 10 Tomás Cohen Arazi 2016-06-28 12:02:33 UTC
The koha-rebuild-zebra script skips if the daemon is enabled.
Comment 11 Magnus Enger 2016-06-30 08:17:42 UTC
I upgraded my production server last night. It is a beefier server with "just" 35 instances. No sign of anything getting stressed out. I'll conclude that the problem with the test server was too many instances, not enough resources and mark this as WORKSFORME. Bug 16823 remains open, though.