We've had an instance with 400k indexing tasks (I'm still unsure if it was an import, but probably) and the indexer daemon kept dying because the DB query we use doesn't put any limit, and the Amazon DB the instance is using was timing out after 40 seconds. It should really use the (passed or default) batch size when querying the DB.
Created attachment 181227 [details] [review] Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size This patch makes the indexer poll the DB in configurable batches. Pretty much like Rabbit does. The only change is that it will now pass a LIMIT <batch_size> to the query. To test: 0. Have KTD started with ES (skip if you have an instance running): $ ktd --es7 up -d 1. Stop the indexer and rabbit: $ ktd --shell k$ sudo koha-es-indexer --stop kohadev k$ sudo service rabbitmq-server stop 2. On a separate terminal, watch the queue: k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev' => SUCCESS: It should be empty 3. Add a bunch of things to index (this works with default KTD data): k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done 4. Start the ES indexer: k$ sudo koha-es-indexer --start kohadev 5. Look at the queue size => FAIL: It shrinks too fast [1] 6. Apply this patch 7. Restart the indexer k$ perl misc/workers/es_indexer_daemon.pl & 8. Repeat 3 9. Look at the queue size: => SUCCESS: It shrinks 'slowly' 10. Sign off :-D [1] This is a problem in certain scenarios, not in all of them. This is the best I come up with in order to test the behavior change. Anyone can set a new size of their taste. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I attempted to test. After steps 8 and 9, the queue goes down slowly. However, I get this repeated message: One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 144.
Created attachment 182176 [details] [review] Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size This patch makes the indexer poll the DB in configurable batches. Pretty much like Rabbit does. The only change is that it will now pass a LIMIT <batch_size> to the query. To test: 0. Have KTD started with ES (skip if you have an instance running): $ ktd --es7 up -d 1. Stop the indexer and rabbit: $ ktd --shell k$ sudo koha-es-indexer --stop kohadev k$ sudo service rabbitmq-server stop 2. On a separate terminal, watch the queue: k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev' => SUCCESS: It should be empty 3. Add a bunch of things to index (this works with default KTD data): k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done 4. Start the ES indexer: k$ sudo koha-es-indexer --start kohadev 5. Look at the queue size => FAIL: It shrinks too fast [1] 6. Apply this patch 7. Restart the indexer k$ perl misc/workers/es_indexer_daemon.pl & 8. Repeat 3 9. Look at the queue size: => SUCCESS: It shrinks 'slowly' 10. Sign off :-D [1] This is a problem in certain scenarios, not in all of them. This is the best I come up with in order to test the behavior change. Anyone can set a new size of their taste. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Trivial and clear patch.. Passing QA
Pushed for 25.05! Well done everyone, thank you!