We have found that in production, when loading large record sets, we can sometimes overload the ES server and cause timeouts. One possible cause here is the cost of generating facets (aggregations) when looking for matching records. We don't use the values here, so shouldn't request them
Created attachment 177655 [details] [review] Bug 39070: Don't request facets when searching for recrod matches This patch adds a new option to Elasticsearch to skip facets in build_query and build_query_compat. There should be no behavior change, however, queries to ES should be simpler To test: 1 - Export some record from Koha 2 - Stage the records for import, matching on 999c 3 - Confirm expected matches 4 - Switch batch to use ISBN for matching 5 - Confirm expected matches 6 - Apply patches, restart all 7 - Switch batch to use 999c 8 - Confirm matches 9 - Switch batch to use ISBN 10 - Confirm matches
Makes a whole lot of sense!
Created attachment 177656 [details] [review] Bug 39070: Don't request facets when searching for recrod matches This patch adds a new option to Elasticsearch to skip facets in build_query and build_query_compat. There should be no behavior change, however, queries to ES should be simpler To test: 1 - Export some record from Koha 2 - Stage the records for import, matching on 999c 3 - Confirm expected matches 4 - Switch batch to use ISBN for matching 5 - Confirm expected matches 6 - Apply patches, restart all 7 - Switch batch to use 999c 8 - Confirm matches 9 - Switch batch to use ISBN 10 - Confirm matches Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
I agree with the approach. There's a lot of improvements to make in the area to avoid hitting ES too much. I followed the test plan ('Elasticsearch' needs to be selected). QA tools happy.
Created attachment 177715 [details] [review] Bug 39070: Don't request facets when searching for recrod matches This patch adds a new option to Elasticsearch to skip facets in build_query and build_query_compat. There should be no behavior change, however, queries to ES should be simpler To test: 1 - Export some record from Koha 2 - Stage the records for import, matching on 999c 3 - Confirm expected matches 4 - Switch batch to use ISBN for matching 5 - Confirm expected matches 6 - Apply patches, restart all 7 - Switch batch to use 999c 8 - Confirm matches 9 - Switch batch to use ISBN 10 - Confirm matches Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io> Signed-off-by: David Cook <dcook@prosentient.com.au>
Pushed for 25.05! Well done everyone, thank you!