Bug 39070

Summary: Elasticsearch facets are not used/needed when finding record matches
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: ToolsAssignee: Nick Clemens (kidclamp) <nick>
Status: Pushed to main --- QA Contact: David Cook <dcook>
Severity: normal    
Priority: P5 - low CC: dcook, mnero, tomascohen
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: Small patch Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
25.05.00
Circulation function:
Attachments: Bug 39070: Don't request facets when searching for recrod matches
Bug 39070: Don't request facets when searching for recrod matches
Bug 39070: Don't request facets when searching for recrod matches

Description Nick Clemens (kidclamp) 2025-02-07 19:23:54 UTC
We have found that in production, when loading large record sets, we can sometimes overload the ES server and cause timeouts.

One possible cause here is the cost of generating facets (aggregations) when looking for matching records.

We don't use the values here, so shouldn't request them
Comment 1 Nick Clemens (kidclamp) 2025-02-07 19:27:19 UTC
Created attachment 177655 [details] [review]
Bug 39070: Don't request facets when searching for recrod matches

This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.

There should be no behavior change, however, queries to ES should be simpler

To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches
Comment 2 Tomás Cohen Arazi (tcohen) 2025-02-07 19:34:49 UTC
Makes a whole lot of sense!
Comment 3 Tomás Cohen Arazi (tcohen) 2025-02-07 20:07:13 UTC
Created attachment 177656 [details] [review]
Bug 39070: Don't request facets when searching for recrod matches

This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.

There should be no behavior change, however, queries to ES should be simpler

To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 4 Tomás Cohen Arazi (tcohen) 2025-02-07 20:14:20 UTC
I agree with the approach. There's a lot of improvements to make in the area to avoid hitting ES too much.

I followed the test plan ('Elasticsearch' needs to be selected). QA tools happy.
Comment 5 David Cook 2025-02-10 23:49:55 UTC
Created attachment 177715 [details] [review]
Bug 39070: Don't request facets when searching for recrod matches

This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.

There should be no behavior change, however, queries to ES should be simpler

To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Comment 6 Katrin Fischer 2025-02-13 15:20:43 UTC
Pushed for 25.05!

Well done everyone, thank you!