Bug 39694 - `es_indexer_daemon.pl` doesn't use batch_size in DB poll mode
Summary: `es_indexer_daemon.pl` doesn't use batch_size in DB poll mode
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low major
Assignee: Tomás Cohen Arazi (tcohen)
QA Contact: Martin Renvoize (ashimema)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-04-21 18:13 UTC by Tomás Cohen Arazi (tcohen)
Modified: 2025-05-12 16:32 UTC (History)
5 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
25.05.00
Circulation function:


Attachments
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size (2.79 KB, patch)
2025-04-21 18:38 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size (2.85 KB, patch)
2025-05-09 13:58 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Tomás Cohen Arazi (tcohen) 2025-04-21 18:13:45 UTC
We've had an instance with 400k indexing tasks (I'm still unsure if it was an import, but probably) and the indexer daemon kept dying because the DB query we use doesn't put any limit, and the Amazon DB the instance is using was timing out after 40 seconds.

It should really use the (passed or default) batch size when querying the DB.
Comment 1 Tomás Cohen Arazi (tcohen) 2025-04-21 18:38:41 UTC
Created attachment 181227 [details] [review]
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size

This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.

The only change is that it will now pass a LIMIT <batch_size> to the
query.

To test:
0. Have KTD started with ES (skip if you have an instance running):
   $ ktd --es7 up -d
1. Stop the indexer and rabbit:
   $ ktd --shell
  k$ sudo koha-es-indexer --stop kohadev
  k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
  k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
  k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
  k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
  k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D

[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 2 David Nind 2025-04-25 21:55:30 UTC
I attempted to test.

After steps 8 and 9, the queue goes down slowly. However, I get this repeated message:

  One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 144.
Comment 3 Martin Renvoize (ashimema) 2025-05-09 13:58:16 UTC
Created attachment 182176 [details] [review]
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size

This patch makes the indexer poll the DB in configurable batches. Pretty
much like Rabbit does.

The only change is that it will now pass a LIMIT <batch_size> to the
query.

To test:
0. Have KTD started with ES (skip if you have an instance running):
   $ ktd --es7 up -d
1. Stop the indexer and rabbit:
   $ ktd --shell
  k$ sudo koha-es-indexer --stop kohadev
  k$ sudo service rabbitmq-server stop
2. On a separate terminal, watch the queue:
  k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev'
=> SUCCESS: It should be empty
3. Add a bunch of things to index (this works with default KTD data):
  k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done
4. Start the ES indexer:
  k$ sudo koha-es-indexer --start kohadev
5. Look at the queue size
=> FAIL: It shrinks too fast [1]
6. Apply this patch
7. Restart the indexer
  k$ perl misc/workers/es_indexer_daemon.pl &
8. Repeat 3
9. Look at the queue size:
=> SUCCESS: It shrinks 'slowly'
10. Sign off :-D

[1] This is a problem in certain scenarios, not in all of them. This is
the best I come up with in order to test the behavior change. Anyone can
set a new size of their taste.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 4 Martin Renvoize (ashimema) 2025-05-09 13:58:34 UTC
Trivial and clear patch.. Passing QA
Comment 5 Katrin Fischer 2025-05-12 16:32:34 UTC
Pushed for 25.05!

Well done everyone, thank you!