At the moment, each instance of Koha on a server gets its own rebuild_zebra.pl daemon which uses approximately 170MB of RAM when idling. While this is not a huge amount on its own, it does not scale well when you start adding many Koha instances on the server. It would be very possible to have 10GB of RAM taken up on a server by idle rebuild_zebra.pl processes alone. In 2018, I wrote a custom Koha indexer daemon (on a non-Debian based OS) which only uses 12MB of RAM. It gets a list of Koha instances, queries the zebraqueue table in those instances, and then forks child processes to run rebuild_zebra.pl (in non-daemon mode) for Koha instances that need it. It also uses advisory file locking, so that only 1 rebuild_zebra.pl process will run per instance at any given time. I would need to refactor the daemon to run using the Debian Koha tooling, but I think that this would be a useful addition to Koha, especially for vendors and other large organisations running multiple Koha instances.
At the moment, it's a single threaded single Perl process that does blocking I/O, so the time it takes to query all Koha databases will degrade as you add Koha databases. While the database queries are fast, this is a potential limitation. However, I could use Mojo to do non-blocking I/O for the database queries, which would remove that potential scalability limitation. Alternatively, I could look at benchmarking the database queries to see if it really is just a hypothetical limitation. It is possible that I could get some paid work time to do this work, but I want to see what community feedback I can get before I do a POC.
Another idea is to use the RabbitMQ. Instead of polling the database via a daemon, I could enhance Koha to send RabbitMQ messages when there is data to index. The daemon would just be subscribed to a Koha index queue and it could trigger indexing jobs as I proposed above.
While these ideas are in the name of efficiency and scalability, there is a downside. They are based on the idea that all the relevant Koha instances are collocated on the same server as the Zebra indexing daemon. Another idea would be to have 1 process per Koha instance (with very minimal memory usage) that either polls the database or listens to RabbitMQ and then forks a child process to run rebuild_zebra.pl (or triggers an API call that does the indexing). This idea is probably the most container friendly. (Another indexing optimization would be to have rebuild_zebra.pl do its work on SSD or RAM disk, since it can perform a lot of I/O for temporarily held data which can be really slow. In the past, we've noticed huge speed improvements by having rebuild_zebra.pl work on a RAM disk.) Reference material: Reading https://stackoverflow.com/questions/9733146/tips-for-keeping-perl-memory-usage-low/9734699#9734699 led to many interesting thoughts on memory usage in Perl, especially the part about garbage collection not working the way I expected in Perl.
(In reply to David Cook from comment #3) > Another idea would be to have 1 process per Koha instance (with very minimal > memory usage) that either polls the database or listens to RabbitMQ and then > forks a child process to run rebuild_zebra.pl (or triggers an API call that > does the indexing). > > This idea is probably the most container friendly. > That said, 60 minimal processes will use more CPU than 1 minimal process, but... that usage should be minimal and non-sustained.
By the way, this would be sponsored by Prosentient Systems, so this isn't a personal project. If people are happy with this idea, I will set aside a work day (or more likely parts of a few work days) and do this work.
Created attachment 114884 [details] [review] Bug 27267: Add memory efficient Zebra indexing daemon This patch adds an alternate Zebra indexing daemon which is much more memory efficient than the current default daemon. (For instance, zebra_indexer.pl uses 6MB of RAM vs rebuild_zebra.pl which uses about 180MB of RAM out of the box while idle.) To test: 0. Apply patch 1. koha-indexer --stop kohadev 2. vi /etc/default/koha-common 3. Update ALTERNATE_INDEXER_DAEMON to be "/kohadevbox/koha/misc/zebra_indexer.pl" 4. koha-indexer --start kohadev 5. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 6. Change 245$a to "Gone" instead of "Gairm" and save the record 7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Gairm 8. Repeat Step 7 for 5+ seconds until your search returns no results 9. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 10. Change 245$a from "Gone" to "Gairm" 11. Repeat Step 7 for 5+ seconds until your search redirects you to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29&found1=1 To confirm lower memory usage: 12. ps -efww | grep "zebra_indexer" 13. top -p <PID of zebra_indexer.pl process> 14. The top tool should show about 6.1MB usage.
It's worth noting that I've used a different model in the patch than what I described in the description. In this case, I'm using the existing Debian Koha indexer infrastructure. The only difference is with the Zebra Indexer daemon that runs. So each Koha instance will still get its own dedicated indexer daemon. The difference is that the indexer daemon will use 6MB of RAM instead of 170-180MB of RAM. If you scale up to 50 Koha libraries on a server, that's 300MB used instead of 9GB for an idle indexer daemon.
I'm loving this change. I have an AWS server with 4GB RAM and only 3 instances. I switched over to using this alternative indexer and I saved 500MB immediately while idling.
Created attachment 117437 [details] [review] Bug 27267: Add memory efficient Zebra indexing daemon This patch adds an alternate Zebra indexing daemon which is much more memory efficient than the current default daemon. (For instance, zebra_indexer.pl uses 6MB of RAM vs rebuild_zebra.pl which uses about 180MB of RAM out of the box while idle.) To test: 0. Apply patch 1. koha-indexer --stop kohadev 2. vi /etc/default/koha-common 3. Update ALTERNATE_INDEXER_DAEMON to be "/kohadevbox/koha/misc/zebra_indexer.pl" 4. koha-indexer --start kohadev 5. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 6. Change 245$a to "Gone" instead of "Gairm" and save the record 7. Go to http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=Gairm 8. Repeat Step 7 for 5+ seconds until your search returns no results 9. Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=29 10. Change 245$a from "Gone" to "Gairm" 11. Repeat Step 7 for 5+ seconds until your search redirects you to http://localhost:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=29&found1=1 To confirm lower memory usage: 12. ps -efww | grep "zebra_indexer" 13. top -p <PID of zebra_indexer.pl process> 14. The top tool should show about 6.1MB usage. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
This works well and is a nice clean implementation. There is one minor QA script failure.. you open 'my $child_in' in the open3 call, but never actually use it... this causes some noise in our QA script understandably but I'm not sure it's a real issue in reality. Signing off
I would be bit hesitant to take this into koha because it seems like this will break very easily due to not using any of the abstractions we have. Also, there is no license header.
(In reply to Joonas Kylmälä from comment #11) > I would be bit hesitant to take this into koha because it seems like this > will break very easily due to not using any of the abstractions we have. > Also, there is no license header. Also it is not just that would break but it also increases now amount of work needed to change koha conf related code and zebra indexing code because you cannot change it only in the common abstraction but the changes are spread out to many places.
> 14. The top tool should show about 6.1MB usage. I get 6.9MB, and there is an additional 17.6MB used by the child process every 5 seconds. So that's 24.5MB taken only to check if there is records to index. Still better than 180MB, but worth noting IMO. Talking about that, I wonder if it's really better to use a child process in this case. Instead of just staying in memory, the code will be loaded/unloaded every 5 seconds. Maybe it will use less memory 99% of the time, but it will probably use more cpu time (and it will need all the memory once every 5 second anyway, so that memory is not really "free") I tried different things to measure memory consumption depending on what we use: - if I add 'require C4::Context' to _get_count (and nothing else), the child process memory consumption goes up to 65MB - if I add 'C4::Context->dbh()', memory consumption goes up to 98MB I don't know exactly where the problem is (though I suspect the DBIx::Class schema to take a significant part of memory), but it would be useful for such cases to have a method that returns a DBI object without loading half of Koha. Also I agree with Joonas, the "$dbh creation" code should not be duplicated.
(In reply to Joonas Kylmälä from comment #12) > (In reply to Joonas Kylmälä from comment #11) > > I would be bit hesitant to take this into koha because it seems like this > > will break very easily due to not using any of the abstractions we have. > > Also, there is no license header. > > Also it is not just that would break but it also increases now amount of > work needed to change koha conf related code and zebra indexing code because > you cannot change it only in the common abstraction but the changes are > spread out to many places. I think that you might misunderstand how this works. The only change is 1 line in /etc/default/koha-common. I've been running it in production for 6 months across a huge number of instances, and it's been going great.
(In reply to Julian Maurice from comment #13) > > 14. The top tool should show about 6.1MB usage. > > I get 6.9MB, and there is an additional 17.6MB used by the child process > every 5 seconds. So that's 24.5MB taken only to check if there is records to > index. Still better than 180MB, but worth noting IMO. > Talking about that, I wonder if it's really better to use a child process in > this case. Instead of just staying in memory, the code will be > loaded/unloaded every 5 seconds. Maybe it will use less memory 99% of the > time, but it will probably use more cpu time (and it will need all the > memory once every 5 second anyway, so that memory is not really "free") We could add an option. Personally, I rather not have the memory reserved by a process which is doing a little check. I much rather have it compete and allow more important processes to reserve memory. We could add an option to do the do the indexing check in the same process rather than using a child process, if you prefer. > I tried different things to measure memory consumption depending on what we > use: > - if I add 'require C4::Context' to _get_count (and nothing else), the child > process memory consumption goes up to 65MB > - if I add 'C4::Context->dbh()', memory consumption goes up to 98MB > > I don't know exactly where the problem is (though I suspect the DBIx::Class > schema to take a significant part of memory), but it would be useful for > such cases to have a method that returns a DBI object without loading half > of Koha. I 100% agree that it would be useful to have a method that returns a DBI object without loading half of Koha. > Also I agree with Joonas, the "$dbh creation" code should not be duplicated. In theory, yes. In practice, we're not at that point. I'm happy for this to stay as Failed QA, until someone else provides that minimal database connection object.
That minimal object could be useful for updatedatabase.pl in Bug 26596 too...
(In reply to Joonas Kylmälä from comment #12) > (In reply to Joonas Kylmälä from comment #11) > > I would be bit hesitant to take this into koha because it seems like this > > will break very easily due to not using any of the abstractions we have. > > Also, there is no license header. After reading through Julian's comments, I think that I understand yours better. The abstractions are too resource intensive to be practical in this case. But I will admit that my code is not ideal. It's practical but does add some potential technical debt. A compromise would be to add core Koha methods that have minimal dependencies but are easier to maintain. I think that I'll just maintain this custom indexer locally though. I don't have time to add those core Koha methods at this point.
David, can you take a look at bug 28306 ? I think you can replace your _get_dbh call by a call to Koha::Database->dbh without using much more memory.
(In reply to Julian Maurice from comment #18) > David, can you take a look at bug 28306 ? > I think you can replace your _get_dbh call by a call to Koha::Database->dbh > without using much more memory. Thanks for commenting here. I saw bug 28306 and thought it would be perfect to use here.
I need to revisit this one at some point. I had forgotten to enable it on one of my servers, and I just saved so much reserved memory as a result.