Bug 39070 - Elasticsearch facets are not used/needed when finding record matches
Summary: Elasticsearch facets are not used/needed when finding record matches
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Nick Clemens (kidclamp)
QA Contact: David Cook
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-02-07 19:23 UTC by Nick Clemens (kidclamp)
Modified: 2025-02-13 15:20 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
25.05.00
Circulation function:


Attachments
Bug 39070: Don't request facets when searching for recrod matches (6.05 KB, patch)
2025-02-07 19:27 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39070: Don't request facets when searching for recrod matches (6.11 KB, patch)
2025-02-07 20:07 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 39070: Don't request facets when searching for recrod matches (6.15 KB, patch)
2025-02-10 23:49 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2025-02-07 19:23:54 UTC
We have found that in production, when loading large record sets, we can sometimes overload the ES server and cause timeouts.

One possible cause here is the cost of generating facets (aggregations) when looking for matching records.

We don't use the values here, so shouldn't request them
Comment 1 Nick Clemens (kidclamp) 2025-02-07 19:27:19 UTC
Created attachment 177655 [details] [review]
Bug 39070: Don't request facets when searching for recrod matches

This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.

There should be no behavior change, however, queries to ES should be simpler

To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches
Comment 2 Tomás Cohen Arazi (tcohen) 2025-02-07 19:34:49 UTC
Makes a whole lot of sense!
Comment 3 Tomás Cohen Arazi (tcohen) 2025-02-07 20:07:13 UTC
Created attachment 177656 [details] [review]
Bug 39070: Don't request facets when searching for recrod matches

This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.

There should be no behavior change, however, queries to ES should be simpler

To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 4 Tomás Cohen Arazi (tcohen) 2025-02-07 20:14:20 UTC
I agree with the approach. There's a lot of improvements to make in the area to avoid hitting ES too much.

I followed the test plan ('Elasticsearch' needs to be selected). QA tools happy.
Comment 5 David Cook 2025-02-10 23:49:55 UTC
Created attachment 177715 [details] [review]
Bug 39070: Don't request facets when searching for recrod matches

This patch adds a new option to Elasticsearch to skip facets in build_query and
build_query_compat.

There should be no behavior change, however, queries to ES should be simpler

To test:
1 - Export some record from Koha
2 - Stage the records for import, matching on 999c
3 - Confirm expected matches
4 - Switch batch to use ISBN for matching
5 - Confirm expected matches
6 - Apply patches, restart all
7 - Switch batch to use 999c
8 - Confirm matches
9 - Switch batch to use ISBN
10 - Confirm matches

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: David Cook <dcook@prosentient.com.au>
Comment 6 Katrin Fischer 2025-02-13 15:20:43 UTC
Pushed for 25.05!

Well done everyone, thank you!