Summary: | `es_indexer_daemon.pl` doesn't use batch_size in DB poll mode | ||
---|---|---|---|
Product: | Koha | Reporter: | Tomás Cohen Arazi (tcohen) <tomascohen> |
Component: | Command-line Utilities | Assignee: | Tomás Cohen Arazi (tcohen) <tomascohen> |
Status: | Pushed to main --- | QA Contact: | Martin Renvoize (ashimema) <martin.renvoize> |
Severity: | major | ||
Priority: | P5 - low | CC: | david, martin.renvoize, nick, robin, tomascohen |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | Trivial patch | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: |
25.05.00
|
Circulation function: | |
Attachments: |
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size |
Description
Tomás Cohen Arazi (tcohen)
2025-04-21 18:13:45 UTC
Created attachment 181227 [details] [review] Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size This patch makes the indexer poll the DB in configurable batches. Pretty much like Rabbit does. The only change is that it will now pass a LIMIT <batch_size> to the query. To test: 0. Have KTD started with ES (skip if you have an instance running): $ ktd --es7 up -d 1. Stop the indexer and rabbit: $ ktd --shell k$ sudo koha-es-indexer --stop kohadev k$ sudo service rabbitmq-server stop 2. On a separate terminal, watch the queue: k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev' => SUCCESS: It should be empty 3. Add a bunch of things to index (this works with default KTD data): k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done 4. Start the ES indexer: k$ sudo koha-es-indexer --start kohadev 5. Look at the queue size => FAIL: It shrinks too fast [1] 6. Apply this patch 7. Restart the indexer k$ perl misc/workers/es_indexer_daemon.pl & 8. Repeat 3 9. Look at the queue size: => SUCCESS: It shrinks 'slowly' 10. Sign off :-D [1] This is a problem in certain scenarios, not in all of them. This is the best I come up with in order to test the behavior change. Anyone can set a new size of their taste. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> I attempted to test. After steps 8 and 9, the queue goes down slowly. However, I get this repeated message: One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 144. Created attachment 182176 [details] [review] Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size This patch makes the indexer poll the DB in configurable batches. Pretty much like Rabbit does. The only change is that it will now pass a LIMIT <batch_size> to the query. To test: 0. Have KTD started with ES (skip if you have an instance running): $ ktd --es7 up -d 1. Stop the indexer and rabbit: $ ktd --shell k$ sudo koha-es-indexer --stop kohadev k$ sudo service rabbitmq-server stop 2. On a separate terminal, watch the queue: k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev' => SUCCESS: It should be empty 3. Add a bunch of things to index (this works with default KTD data): k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done 4. Start the ES indexer: k$ sudo koha-es-indexer --start kohadev 5. Look at the queue size => FAIL: It shrinks too fast [1] 6. Apply this patch 7. Restart the indexer k$ perl misc/workers/es_indexer_daemon.pl & 8. Repeat 3 9. Look at the queue size: => SUCCESS: It shrinks 'slowly' 10. Sign off :-D [1] This is a problem in certain scenarios, not in all of them. This is the best I come up with in order to test the behavior change. Anyone can set a new size of their taste. Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> Trivial and clear patch.. Passing QA Pushed for 25.05! Well done everyone, thank you! |