Bug 40853 - ElasticsearchBoostFieldMatch - needs to boost results more
Summary: ElasticsearchBoostFieldMatch - needs to boost results more
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Nick Clemens (kidclamp)
QA Contact: Thomas Klausner
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-09-22 18:47 UTC by Nick Clemens (kidclamp)
Modified: 2025-10-08 14:46 UTC (History)
7 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference (5.31 KB, patch)
2025-10-03 19:25 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 40853: Unit tests (2.43 KB, patch)
2025-10-03 19:35 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference (5.36 KB, patch)
2025-10-06 20:32 UTC, Jason Robb
Details | Diff | Splinter Review
Bug 40853: Unit tests (2.47 KB, patch)
2025-10-06 20:32 UTC, Jason Robb
Details | Diff | Splinter Review
MARC records for testing (2.38 MB, application/marc)
2025-10-08 14:46 UTC, Jason Robb
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2025-09-22 18:47:54 UTC
When searching for book titles that may have words in many fields, e.g. 'book of hope', the boost is not enough, especially when other fields like 'title' are boosted.

In production with title boost of 32 the general results are getting a score of ~600
The boost query is getting a score of ~10 - not enough to boost past the general results

We can try a few things:
1 - Use the 'boost' field from 'title-cover' matching in the extra query
2 - Add a system preference (or hard code) to define how much to boost the additional query by
3 - Try using a boost query rather than a bool query: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-boosting-query
Comment 1 Nick Clemens (kidclamp) 2025-10-03 19:25:05 UTC
Created attachment 187422 [details] [review]
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference

This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch.
You can add any positive number, or a number between 0 and 1 for negative boost

To test:
1 - Add a record with
    245 $a novel
2 - Add a record with
    245 $a A novel : $b about things / $c by me
3 - Search for: novel
4 - Search for: a novel
5 - Apply patch, restart all
6 - Repeat searches, note exact titles are boosted
7 - Test with more titles as you are able
Comment 2 Nick Clemens (kidclamp) 2025-10-03 19:35:59 UTC
Created attachment 187425 [details] [review]
Bug 40853: Unit tests
Comment 3 Jason Robb 2025-10-06 20:32:28 UTC
Created attachment 187500 [details] [review]
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference

This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch.
You can add any positive number, or a number between 0 and 1 for negative boost

To test:
1 - Add a record with
    245 $a novel
2 - Add a record with
    245 $a A novel : $b about things / $c by me
3 - Search for: novel
4 - Search for: a novel
5 - Apply patch, restart all
6 - Repeat searches, note exact titles are boosted
7 - Test with more titles as you are able

Signed-off-by: Jason Robb <jrobb@sekls.org>
Comment 4 Jason Robb 2025-10-06 20:32:31 UTC
Created attachment 187501 [details] [review]
Bug 40853: Unit tests

Signed-off-by: Jason Robb <jrobb@sekls.org>
Comment 5 Jason Robb 2025-10-06 20:42:00 UTC
I tested this with a batch of 1000 records from our production server for a search on "Book of Hope" in ktd and it worked well! My exact title search went from #649 and #651 to #1 and #2.

A few notes of things I ran into:

I tested with es8 first and didn't need the patch -- enabling the boost syspref with es8 worked.

I tested with es7 after that (which is what our current production server is running) and the issue was present; the syspref didn't do any sort of boosting. When I applied the patch and enabled the boost and left the amount at 0 my search results were much better.

I tried playing with the new amount value for the syspref and it didn't seem to do much, but the patch still gave me much better exact title boosting regardless so I went ahead and signed off.
Comment 6 Thomas Klausner 2025-10-08 14:17:20 UTC
I wanted to QA this, but I could not see any difference between before/after applying the patch, and the search result seemed correct before applying.

Maybe the test plan can be improved? For example I have no clue what the expected search results should be for 'novel' / 'a novel' (because the newly created books with those titles where shown at the start of the list even before applying the patch)
Comment 7 Jason Robb 2025-10-08 14:46:19 UTC
Created attachment 187582 [details]
MARC records for testing

This marc file contains 1000 bibs with the terms "book" "of" and "hope" somewhere in the records.

There are two records with the exact title "The book of hope".

To test with these records:

1.) Import the records
2.) Do a keyword search for book of hope (no quotes) and note the exact titles are not near the top of the results
3.) Enable ElasticsearchBoostFieldMatch, do the same search, still not boosted
4.) Apply the patch
5.) Search again with ElasticsearchBoostFieldMatch enabled, the exact titles should be at the top of the results this time

As I noted in another comment, this only happens with es7. When testing with es8 the search does get boosted at step 3.