Bug 27267 - Create more efficient daemon for indexing Zebra
Summary: Create more efficient daemon for indexing Zebra
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Cook
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-12-18 00:27 UTC by David Cook
Modified: 2021-05-10 23:51 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 27267: Add memory efficient Zebra indexing daemon (4.60 KB, patch)
2021-01-06 03:15 UTC, David Cook
Details | Diff | Splinter Review
Bug 27267: Add memory efficient Zebra indexing daemon (4.65 KB, patch)
2021-03-01 11:18 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2020-12-18 00:27:45 UTC
At the moment, each instance of Koha on a server gets its own rebuild_zebra.pl daemon which uses approximately 170MB of RAM when idling. While this is not a huge amount on its own, it does not scale well when you start adding many Koha instances on the server. It would be very possible to have 10GB of RAM taken up on a server by idle rebuild_zebra.pl processes alone.

In 2018, I wrote a custom Koha indexer daemon (on a non-Debian based OS) which only uses 12MB of RAM. It gets a list of Koha instances, queries the zebraqueue table in those instances, and then forks child processes to run rebuild_zebra.pl (in non-daemon mode) for Koha instances that need it. It also uses advisory file locking, so that only 1 rebuild_zebra.pl process will run per instance at any given time.

I would need to refactor the daemon to run using the Debian Koha tooling, but I think that this would be a useful addition to Koha, especially for vendors and other large organisations running multiple Koha instances.
Comment 1 David Cook 2020-12-18 00:30:46 UTC Comment hidden (obsolete)
Comment 2 David Cook 2020-12-18 00:33:40 UTC Comment hidden (obsolete)
Comment 3 David Cook 2020-12-18 00:44:08 UTC Comment hidden (obsolete)
Comment 4 David Cook 2020-12-18 00:50:16 UTC Comment hidden (obsolete)
Comment 5 David Cook 2020-12-18 00:53:09 UTC Comment hidden (obsolete)
Comment 6 David Cook 2021-01-06 03:15:15 UTC
Created attachment 114884 [details] [review]
Bug 27267: Add memory efficient Zebra indexing daemon

This patch adds an alternate Zebra indexing daemon which
is much more memory efficient than the current default daemon.

(For instance, zebra_indexer.pl uses 6MB of RAM vs rebuild_zebra.pl
 which uses about 180MB of RAM out of the box while idle.)

To test:
0. Apply patch
1. koha-indexer --stop kohadev
2. vi /etc/default/koha-common
3. Update ALTERNATE_INDEXER_DAEMON to be "/kohadevbox/koha/misc/zebra_indexer.pl"
4. koha-indexer --start kohadev
5. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29
6. Change 245$a to "Gone" instead of "Gairm" and save the record
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Gairm
8. Repeat Step 7 for 5+ seconds until your search returns no results
9. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29
10. Change 245$a from "Gone" to "Gairm"
11. Repeat Step 7 for 5+ seconds until your search redirects you to
http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29&found1=1

To confirm lower memory usage:
12. ps -efww | grep "zebra_indexer"
13. top -p <PID of zebra_indexer.pl process>
14. The top tool should show about 6.1MB usage.
Comment 7 David Cook 2021-01-06 03:20:27 UTC
It's worth noting that I've used a different model in the patch than what I described in the description.

In this case, I'm using the existing Debian Koha indexer infrastructure. The only difference is with the Zebra Indexer daemon that runs. So each Koha instance will still get its own dedicated indexer daemon. 

The difference is that the indexer daemon will use 6MB of RAM instead of 170-180MB of RAM. 

If you scale up to 50 Koha libraries on a server, that's 300MB used instead of 9GB for an idle indexer daemon.
Comment 8 David Cook 2021-02-17 03:33:43 UTC
I'm loving this change.

I have an AWS server with 4GB RAM and only 3 instances. I switched over to using this alternative indexer and I saved 500MB immediately while idling.
Comment 9 Martin Renvoize 2021-03-01 11:18:59 UTC
Created attachment 117437 [details] [review]
Bug 27267: Add memory efficient Zebra indexing daemon

This patch adds an alternate Zebra indexing daemon which
is much more memory efficient than the current default daemon.

(For instance, zebra_indexer.pl uses 6MB of RAM vs rebuild_zebra.pl
 which uses about 180MB of RAM out of the box while idle.)

To test:
0. Apply patch
1. koha-indexer --stop kohadev
2. vi /etc/default/koha-common
3. Update ALTERNATE_INDEXER_DAEMON to be "/kohadevbox/koha/misc/zebra_indexer.pl"
4. koha-indexer --start kohadev
5. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29
6. Change 245$a to "Gone" instead of "Gairm" and save the record
7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Gairm
8. Repeat Step 7 for 5+ seconds until your search returns no results
9. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29
10. Change 245$a from "Gone" to "Gairm"
11. Repeat Step 7 for 5+ seconds until your search redirects you to
http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29&found1=1

To confirm lower memory usage:
12. ps -efww | grep "zebra_indexer"
13. top -p <PID of zebra_indexer.pl process>
14. The top tool should show about 6.1MB usage.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize 2021-03-01 11:20:37 UTC
This works well and is a nice clean implementation.

There is one minor QA script failure.. you open 'my $child_in' in the open3 call, but never actually use it... this causes some noise in our QA script understandably but I'm not sure it's a real issue in reality.

Signing off
Comment 11 Joonas Kylmälä 2021-04-07 09:38:44 UTC
I would be bit hesitant to take this into koha because it seems like this will break very easily due to not using any of the abstractions we have. Also, there is no license header.
Comment 12 Joonas Kylmälä 2021-04-07 09:40:49 UTC
(In reply to Joonas Kylmälä from comment #11)
> I would be bit hesitant to take this into koha because it seems like this
> will break very easily due to not using any of the abstractions we have.
> Also, there is no license header.

Also it is not just that would break but it also increases now amount of work needed to change koha conf related code and zebra indexing code because you cannot change it only in the common abstraction but the changes are spread out to many places.
Comment 13 Julian Maurice 2021-04-09 14:04:38 UTC
> 14. The top tool should show about 6.1MB usage.

I get 6.9MB, and there is an additional 17.6MB used by the child process every 5 seconds. So that's 24.5MB taken only to check if there is records to index. Still better than 180MB, but worth noting IMO.
Talking about that, I wonder if it's really better to use a child process in this case. Instead of just staying in memory, the code will be loaded/unloaded every 5 seconds. Maybe it will use less memory 99% of the time, but it will probably use more cpu time (and it will need all the memory once every 5 second anyway, so that memory is not really "free")

I tried different things to measure memory consumption depending on what we use:
- if I add 'require C4::Context' to _get_count (and nothing else), the child process memory consumption goes up to 65MB
- if I add 'C4::Context->dbh()', memory consumption goes up to 98MB

I don't know exactly where the problem is (though I suspect the DBIx::Class schema to take a significant part of memory), but it would be useful for such cases to have a method that returns a DBI object without loading half of Koha.

Also I agree with Joonas, the "$dbh creation" code should not be duplicated.
Comment 14 David Cook 2021-04-12 01:16:36 UTC
(In reply to Joonas Kylmälä from comment #12)
> (In reply to Joonas Kylmälä from comment #11)
> > I would be bit hesitant to take this into koha because it seems like this
> > will break very easily due to not using any of the abstractions we have.
> > Also, there is no license header.
> 
> Also it is not just that would break but it also increases now amount of
> work needed to change koha conf related code and zebra indexing code because
> you cannot change it only in the common abstraction but the changes are
> spread out to many places.

I think that you might misunderstand how this works. The only change is 1 line in /etc/default/koha-common.

I've been running it in production for 6 months across a huge number of instances, and it's been going great.
Comment 15 David Cook 2021-04-12 01:24:44 UTC
(In reply to Julian Maurice from comment #13)
> > 14. The top tool should show about 6.1MB usage.
> 
> I get 6.9MB, and there is an additional 17.6MB used by the child process
> every 5 seconds. So that's 24.5MB taken only to check if there is records to
> index. Still better than 180MB, but worth noting IMO.
> Talking about that, I wonder if it's really better to use a child process in
> this case. Instead of just staying in memory, the code will be
> loaded/unloaded every 5 seconds. Maybe it will use less memory 99% of the
> time, but it will probably use more cpu time (and it will need all the
> memory once every 5 second anyway, so that memory is not really "free")
 
We could add an option. Personally, I rather not have the memory reserved by a process which is doing a little check. I much rather have it compete and allow more important processes to reserve memory. 

We could add an option to do the do the indexing check in the same process rather than using a child process, if you prefer. 

> I tried different things to measure memory consumption depending on what we
> use:
> - if I add 'require C4::Context' to _get_count (and nothing else), the child
> process memory consumption goes up to 65MB
> - if I add 'C4::Context->dbh()', memory consumption goes up to 98MB
> 
> I don't know exactly where the problem is (though I suspect the DBIx::Class
> schema to take a significant part of memory), but it would be useful for
> such cases to have a method that returns a DBI object without loading half
> of Koha.
 
I 100% agree that it would be useful to have a method that returns a DBI object without loading half of Koha. 

> Also I agree with Joonas, the "$dbh creation" code should not be duplicated.

In theory, yes. In practice, we're not at that point. 

I'm happy for this to stay as Failed QA, until someone else provides that minimal database connection object.
Comment 16 David Cook 2021-04-12 01:32:31 UTC
That minimal object could be useful for updatedatabase.pl in Bug 26596 too...
Comment 17 David Cook 2021-04-12 01:36:24 UTC
(In reply to Joonas Kylmälä from comment #12)
> (In reply to Joonas Kylmälä from comment #11)
> > I would be bit hesitant to take this into koha because it seems like this
> > will break very easily due to not using any of the abstractions we have.
> > Also, there is no license header.
 
After reading through Julian's comments, I think that I understand yours better.

The abstractions are too resource intensive to be practical in this case. But I will admit that my code is not ideal. It's practical but does add some potential technical debt. A compromise would be to add core Koha methods that have minimal dependencies but are easier to maintain. 

I think that I'll just maintain this custom indexer locally though. I don't have time to add those core Koha methods at this point.
Comment 18 Julian Maurice 2021-05-10 06:57:56 UTC
David, can you take a look at bug 28306 ?
I think you can replace your _get_dbh call by a call to Koha::Database->dbh without using much more memory.
Comment 19 David Cook 2021-05-10 23:51:29 UTC
(In reply to Julian Maurice from comment #18)
> David, can you take a look at bug 28306 ?
> I think you can replace your _get_dbh call by a call to Koha::Database->dbh
> without using much more memory.

Thanks for commenting here. I saw bug 28306 and thought it would be perfect to use here.