Bugzilla – Attachment 182176 Details for
Bug 39694
`es_indexer_daemon.pl` doesn't use batch_size in DB poll mode
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size
Bug-39694-Make-esindexerdaemonpl-use-batchsize-to-.patch (text/plain), 2.85 KB, created by
Martin Renvoize (ashimema)
on 2025-05-09 13:58:16 UTC
(
hide
)
Description:
Bug 39694: Make `es_indexer_daemon.pl` use batch_size to limit DB poll size
Filename:
MIME Type:
Creator:
Martin Renvoize (ashimema)
Created:
2025-05-09 13:58:16 UTC
Size:
2.85 KB
patch
obsolete
>From a7871d959a68aef43ca5a59e972991f9185aef5e Mon Sep 17 00:00:00 2001 >From: Tomas Cohen Arazi <tomascohen@theke.io> >Date: Mon, 21 Apr 2025 15:16:32 -0300 >Subject: [PATCH] Bug 39694: Make `es_indexer_daemon.pl` use batch_size to > limit DB poll size > >This patch makes the indexer poll the DB in configurable batches. Pretty >much like Rabbit does. > >The only change is that it will now pass a LIMIT <batch_size> to the >query. > >To test: >0. Have KTD started with ES (skip if you have an instance running): > $ ktd --es7 up -d >1. Stop the indexer and rabbit: > $ ktd --shell > k$ sudo koha-es-indexer --stop kohadev > k$ sudo service rabbitmq-server stop >2. On a separate terminal, watch the queue: > k$ watch -n 0.5 'echo "SELECT COUNT(*) FROM background_jobs WHERE status=\"new\"" | koha-mysql kohadev' >=> SUCCESS: It should be empty >3. Add a bunch of things to index (this works with default KTD data): > k$ for i in $(seq 1 438; seq 1 438) ; do echo "INSERT INTO background_jobs (status,queue,data) VALUES ('new','elastic_index','{\"record_server\":\"biblioserver\",\"record_ids\":[$i]}')" | koha-mysql kohadev ; done >4. Start the ES indexer: > k$ sudo koha-es-indexer --start kohadev >5. Look at the queue size >=> FAIL: It shrinks too fast [1] >6. Apply this patch >7. Restart the indexer > k$ perl misc/workers/es_indexer_daemon.pl & >8. Repeat 3 >9. Look at the queue size: >=> SUCCESS: It shrinks 'slowly' >10. Sign off :-D > >[1] This is a problem in certain scenarios, not in all of them. This is >the best I come up with in order to test the behavior change. Anyone can >set a new size of their taste. > >Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> >Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk> >--- > misc/workers/es_indexer_daemon.pl | 10 ++++++---- > 1 file changed, 6 insertions(+), 4 deletions(-) > >diff --git a/misc/workers/es_indexer_daemon.pl b/misc/workers/es_indexer_daemon.pl >index aa2f2a9db26..24a54d52f68 100755 >--- a/misc/workers/es_indexer_daemon.pl >+++ b/misc/workers/es_indexer_daemon.pl >@@ -64,7 +64,8 @@ use Koha::BackgroundJobs; > use Koha::SearchEngine; > use Koha::SearchEngine::Indexer; > >-my ( $help, $batch_size ); >+my $help; >+my $batch_size = 10; > > my $not_found_retries = {}; > my $max_retries = $ENV{MAX_RETRIES} || 10; >@@ -76,8 +77,6 @@ GetOptions( > > pod2usage(0) if $help; > >-$batch_size //= 10; >- > warn "Not using Elasticsearch" unless C4::Context->preference('SearchEngine') eq 'Elasticsearch'; > > my $logger = Koha::Logger->get( { interface => 'worker' } ); >@@ -172,7 +171,10 @@ while (1) { > } > > } else { >- @jobs = Koha::BackgroundJobs->search( { status => 'new', queue => 'elastic_index' } )->as_list; >+ @jobs = Koha::BackgroundJobs->search( >+ { status => 'new', queue => 'elastic_index' }, >+ { rows => $batch_size } >+ )->as_list; > commit(@jobs); > @jobs = (); > sleep 10; >-- >2.49.0
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 39694
:
181227
| 182176