Bug 40890

Summary: Make batch_size configurable for koha-es-indexer
Product: Koha Reporter: Tomás Cohen Arazi (tcohen) <tomascohen>
Component: Searching - ElasticsearchAssignee: Tomás Cohen Arazi (tcohen) <tomascohen>
Status: Pushed to main --- QA Contact: Martin Renvoize (ashimema) <martin.renvoize>
Severity: enhancement    
Priority: P5 - low CC: evelyn, kyle, martin.renvoize, mtj, tomascohen, tristin.stagg
Version: unspecified   
Hardware: All   
OS: All   
GIT URL: Initiative type: ---
Sponsorship status: --- Comma delimited list of Sponsors:
Crowdfunding goal: 0 Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
A new configuration entry has been introduced in `koha-conf.xml` to control how many Elasticsearch update tasks the indexer processes per iteration. Previously, the indexer defaulted to a batch size of 10. While the `koha-es-indexer` command supported overriding this with the `--batch_size` parameter, there was no way to set a different default value. This change is particularly beneficial for large sites, where the indexer may otherwise struggle to keep pace with the update queue during peak activity.
Version(s) released in:
25.11.00
Circulation function:
Attachments: Bug 40890: Make batch_size configurable for koha-es-indexer
Bug 40890: Make batch_size configurable for koha-es-indexer
Bug 40890: Move fetching of batch size to within the instance loop
Bug 40890: Make batch_size configurable for koha-es-indexer
Bug 40890: Move fetching of batch size to within the instance loop

Description Tomás Cohen Arazi (tcohen) 2025-09-26 19:49:53 UTC
The script allows setting a custom batch_size e.g. `koha-es-indexer --batch_size 100 --restart library` but it cannot be set permanently. So a reboot or a naive restart will set it back to 10.
Comment 1 Tomás Cohen Arazi (tcohen) 2025-09-26 20:07:05 UTC
Created attachment 187014 [details] [review]
Bug 40890: Make batch_size configurable for koha-es-indexer

This patch adds a new configuration entry in `koha-conf.xml` for making
it possible to set a different `batch_size` for `koha-es-indexer`.

The current implementation has a default of *10* that can be overridden
using the CLI parameter `--batch_size N`.

Unfortunately, if you restart the server or the service without passing
the parameter, it will be set back to 10. This patch solves that by
adding a new config entry and making the script use it. The passed CLI
parameter will keep overridding the configured value.

To test:
1. Run:
   $ ktd --shell
  k$ koha-es-indexer --restart --batch_size 100 kohadev
  k$ ps -ef | grep elastic
=> SUCCESS: No errors. Notice '100' is passed to the called perl script
2. Run:
  k$ koha-es-indexer --restart kohadev
  k$ ps -ef | grep elastic
=> FAIL: Notice '10' is the new value
3. Apply this patch
4. Run:
  k$ reset_all
5. Repeat 1 and 2
=> SUCCESS: '100' is kept.
6. Try standalone `--start` too
7. Sign off :-D
Comment 2 Kyle M Hall (khall) 2025-11-03 19:38:42 UTC
Created attachment 188996 [details] [review]
Bug 40890: Make batch_size configurable for koha-es-indexer

This patch adds a new configuration entry in `koha-conf.xml` for making
it possible to set a different `batch_size` for `koha-es-indexer`.

The current implementation has a default of *10* that can be overridden
using the CLI parameter `--batch_size N`.

Unfortunately, if you restart the server or the service without passing
the parameter, it will be set back to 10. This patch solves that by
adding a new config entry and making the script use it. The passed CLI
parameter will keep overridding the configured value.

To test:
1. Run:
   $ ktd --shell
  k$ koha-es-indexer --restart --batch_size 100 kohadev
  k$ ps -ef | grep elastic
=> SUCCESS: No errors. Notice '100' is passed to the called perl script
2. Run:
  k$ koha-es-indexer --restart kohadev
  k$ ps -ef | grep elastic
=> FAIL: Notice '10' is the new value
3. Apply this patch
4. Run:
  k$ reset_all
5. Repeat 1 and 2
=> SUCCESS: '100' is kept.
6. Try standalone `--start` too
7. Sign off :-D

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 3 Kyle M Hall (khall) 2025-11-03 19:38:51 UTC
Created attachment 188997 [details] [review]
Bug 40890: Move fetching of batch size to within the instance loop

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 4 Martin Renvoize (ashimema) 2025-11-06 07:37:54 UTC
Created attachment 189143 [details] [review]
Bug 40890: Make batch_size configurable for koha-es-indexer

This patch adds a new configuration entry in `koha-conf.xml` for making
it possible to set a different `batch_size` for `koha-es-indexer`.

The current implementation has a default of *10* that can be overridden
using the CLI parameter `--batch_size N`.

Unfortunately, if you restart the server or the service without passing
the parameter, it will be set back to 10. This patch solves that by
adding a new config entry and making the script use it. The passed CLI
parameter will keep overridding the configured value.

To test:
1. Run:
   $ ktd --shell
  k$ koha-es-indexer --restart --batch_size 100 kohadev
  k$ ps -ef | grep elastic
=> SUCCESS: No errors. Notice '100' is passed to the called perl script
2. Run:
  k$ koha-es-indexer --restart kohadev
  k$ ps -ef | grep elastic
=> FAIL: Notice '10' is the new value
3. Apply this patch
4. Run:
  k$ reset_all
5. Repeat 1 and 2
=> SUCCESS: '100' is kept.
6. Try standalone `--start` too
7. Sign off :-D

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 5 Martin Renvoize (ashimema) 2025-11-06 07:37:56 UTC
Created attachment 189144 [details] [review]
Bug 40890: Move fetching of batch size to within the instance loop

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 6 Lucas Gass (lukeg) 2025-11-07 17:56:06 UTC
Nice work everyone!

Pushed to main for 25.11
Comment 7 Evelyn Hartline 2025-11-07 21:02:43 UTC
Could this be backported to 25.05?