It is possible to extract the ids for the resultset without making ES return the records (documents) themselves. This would save processing time in both ends, and memory (building a 5000 docs resultset, serializing it, and deserializing it, needs to take resources).
Created attachment 174115 [details] [review] Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
in order to understand the difference in the behavior, you need an ES-enabled KTD: ```shell ktd --proxy --es7 up -d ktd --shell ``` Then compare the output of the following commands: ```shell curl -X GET 'http://es:9200/koha_kohadev_biblios/_search?scroll=1m&size=10' -H 'Content-Type: application/json' -d'{"query":{"match_all":{}}}' ``` and ```shell curl -X GET 'http://es:9200/koha_kohadev_biblios/_search?scroll=1m&size=10' -H 'Content-Type: application/json' -d'{"_source":false,"query":{"match_all":{}}}' ```
Created attachment 174583 [details] [review] Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Small change, big impact, passing QA
Pushed for 25.05! Well done everyone, thank you!
Moving from 'enhancement' to 'minor' - this is a performance bug really, and should be backported to make the tool more usable
Nice work everyone! Pushed to 24.11.x for 24.11.02