Bug 38386

Summary: compare_es_to_db.pl shouldn't retrieve the records from ES
Product: Koha Reporter: Tomás Cohen Arazi (tcohen) <tomascohen>
Component: Command-line UtilitiesAssignee: Tomás Cohen Arazi (tcohen) <tomascohen>
Status: RESOLVED FIXED QA Contact: Nick Clemens (kidclamp) <nick>
Severity: minor    
Priority: P5 - low CC: dcook, jonathan.druart, martin.renvoize, nick, paul.derscheid, robin, tomascohen, wainuiwitikapark
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: Trivial patch Documentation contact:
Documentation submission: Text to go in the release notes:
This small enhancement makes the `compare_es_to_db.pl` maintenance script require less resources when run.
Version(s) released in:
25.05.00,24.11.02
Circulation function:
Bug Depends on: 22831    
Bug Blocks:    
Attachments: Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES
Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES

Description Tomás Cohen Arazi (tcohen) 2024-11-07 11:13:59 UTC
It is possible to extract the ids for the resultset without making ES return the records (documents) themselves. This would save processing time in both ends, and memory (building a 5000 docs resultset, serializing it, and deserializing it, needs to take resources).
Comment 1 Tomás Cohen Arazi (tcohen) 2024-11-07 11:19:40 UTC
Created attachment 174115 [details] [review]
Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 2 Tomás Cohen Arazi (tcohen) 2024-11-07 11:26:36 UTC
in order to understand the difference in the behavior, you need an ES-enabled KTD:

```shell
ktd --proxy --es7 up -d
ktd --shell
```

Then compare the output of the following commands:

```shell
curl -X GET 'http://es:9200/koha_kohadev_biblios/_search?scroll=1m&size=10' -H 'Content-Type: application/json' -d'{"query":{"match_all":{}}}'
```

and

```shell
curl -X GET 'http://es:9200/koha_kohadev_biblios/_search?scroll=1m&size=10' -H 'Content-Type: application/json' -d'{"_source":false,"query":{"match_all":{}}}'
```
Comment 3 Nick Clemens (kidclamp) 2024-11-15 13:54:07 UTC
Created attachment 174583 [details] [review]
Bug 38386: Make compare_es_to_db.pl not retrieve the records from ES

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 4 Nick Clemens (kidclamp) 2024-11-15 13:56:10 UTC
Small change, big impact, passing QA
Comment 5 Katrin Fischer 2024-12-23 11:23:29 UTC
Pushed for 25.05!

Well done everyone, thank you!
Comment 6 Nick Clemens (kidclamp) 2025-02-04 16:51:04 UTC
Moving from 'enhancement' to 'minor' - this is a performance bug really, and should be backported to make the tool more usable
Comment 7 Paul Derscheid 2025-03-22 11:44:32 UTC
Nice work everyone! 

Pushed to 24.11.x for 24.11.02
Comment 8 Wainui Witika-Park 2025-04-10 04:48:41 UTC
Not backporting to 24.05.x unless requested
Comment 9 Tomás Cohen Arazi (tcohen) 2025-04-10 18:10:59 UTC
I think this should be backported if possible. It can really stress too much the ES without it, and is really simple.