Bug 40853

Summary: ElasticsearchBoostFieldMatch - needs to boost results more
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens (kidclamp) <nick>
Status: Signed Off --- QA Contact: Thomas Klausner <domm>
Severity: normal    
Priority: P5 - low CC: ahernandez, caroline.cyr-la-rose, Chip.Halvorsen, domm, esther.melander, jrobb, michaela.sieber, mspinney
Version: Main   
Hardware: All   
OS: All   
GIT URL: Initiative type: ---
Sponsorship status: --- Comma delimited list of Sponsors:
Crowdfunding goal: 0 Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference
Bug 40853: Unit tests
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference
Bug 40853: Unit tests
MARC records for testing

Description Nick Clemens (kidclamp) 2025-09-22 18:47:54 UTC
When searching for book titles that may have words in many fields, e.g. 'book of hope', the boost is not enough, especially when other fields like 'title' are boosted.

In production with title boost of 32 the general results are getting a score of ~600
The boost query is getting a score of ~10 - not enough to boost past the general results

We can try a few things:
1 - Use the 'boost' field from 'title-cover' matching in the extra query
2 - Add a system preference (or hard code) to define how much to boost the additional query by
3 - Try using a boost query rather than a bool query: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-boosting-query
Comment 1 Nick Clemens (kidclamp) 2025-10-03 19:25:05 UTC
Created attachment 187422 [details] [review]
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference

This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch.
You can add any positive number, or a number between 0 and 1 for negative boost

To test:
1 - Add a record with
    245 $a novel
2 - Add a record with
    245 $a A novel : $b about things / $c by me
3 - Search for: novel
4 - Search for: a novel
5 - Apply patch, restart all
6 - Repeat searches, note exact titles are boosted
7 - Test with more titles as you are able
Comment 2 Nick Clemens (kidclamp) 2025-10-03 19:35:59 UTC
Created attachment 187425 [details] [review]
Bug 40853: Unit tests
Comment 3 Jason Robb 2025-10-06 20:32:28 UTC
Created attachment 187500 [details] [review]
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference

This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch.
You can add any positive number, or a number between 0 and 1 for negative boost

To test:
1 - Add a record with
    245 $a novel
2 - Add a record with
    245 $a A novel : $b about things / $c by me
3 - Search for: novel
4 - Search for: a novel
5 - Apply patch, restart all
6 - Repeat searches, note exact titles are boosted
7 - Test with more titles as you are able

Signed-off-by: Jason Robb <jrobb@sekls.org>
Comment 4 Jason Robb 2025-10-06 20:32:31 UTC
Created attachment 187501 [details] [review]
Bug 40853: Unit tests

Signed-off-by: Jason Robb <jrobb@sekls.org>
Comment 5 Jason Robb 2025-10-06 20:42:00 UTC
I tested this with a batch of 1000 records from our production server for a search on "Book of Hope" in ktd and it worked well! My exact title search went from #649 and #651 to #1 and #2.

A few notes of things I ran into:

I tested with es8 first and didn't need the patch -- enabling the boost syspref with es8 worked.

I tested with es7 after that (which is what our current production server is running) and the issue was present; the syspref didn't do any sort of boosting. When I applied the patch and enabled the boost and left the amount at 0 my search results were much better.

I tried playing with the new amount value for the syspref and it didn't seem to do much, but the patch still gave me much better exact title boosting regardless so I went ahead and signed off.
Comment 6 Thomas Klausner 2025-10-08 14:17:20 UTC
I wanted to QA this, but I could not see any difference between before/after applying the patch, and the search result seemed correct before applying.

Maybe the test plan can be improved? For example I have no clue what the expected search results should be for 'novel' / 'a novel' (because the newly created books with those titles where shown at the start of the list even before applying the patch)
Comment 7 Jason Robb 2025-10-08 14:46:19 UTC
Created attachment 187582 [details]
MARC records for testing

This marc file contains 1000 bibs with the terms "book" "of" and "hope" somewhere in the records.

There are two records with the exact title "The book of hope".

To test with these records:

1.) Import the records
2.) Do a keyword search for book of hope (no quotes) and note the exact titles are not near the top of the results
3.) Enable ElasticsearchBoostFieldMatch, do the same search, still not boosted
4.) Apply the patch
5.) Search again with ElasticsearchBoostFieldMatch enabled, the exact titles should be at the top of the results this time

As I noted in another comment, this only happens with es7. When testing with es8 the search does get boosted at step 3.
Comment 8 Nick Clemens (kidclamp) 2025-10-20 19:02:20 UTC
(In reply to Thomas Klausner from comment #6)
> I wanted to QA this, but I could not see any difference between before/after
> applying the patch, and the search result seemed correct before applying.
> 
> Maybe the test plan can be improved? For example I have no clue what the
> expected search results should be for 'novel' / 'a novel' (because the newly
> created books with those titles where shown at the start of the list even
> before applying the patch)

Try adding weights to fields before applying the patch as well:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl


Set weights like:
title 32
author 16
subject 8
Comment 9 Caroline Cyr La Rose 2025-10-23 21:08:22 UTC
I tried this a little bit. I used our mappings, which have weights
author = 5.00
subject = 10.00
title = 20.00
title-cover = 25.00 
(I think the title-cover one is there because there was no ElasticsearchBoostFieldMatch)
title-series = 15.00


I imported 100 records which have the word "angelique" (or related word like angel, etc). There is, among others
245$a Angélique / $c Guillaume Musso 
245$a Angélique : $b roman / $c Guillaume Musso. 
245$a Angelique. $n 1 $p Buffalo Hunt $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley].
245$a Angelique. $n 3 $p Autumn alone $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley].
245$a Angelique. $n 2 $p The long way home $c Statement of responsibility, etc. Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 
245$a Angelique. $n 4 $p Angel in the snow $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 

I did these tests on KTD with ES8.I simply search for angelique in the bar at the top of the staff interface.

With just the weights and no prefs, the results are like this
"Angelique. 4 Angel in the snow" 16th result
"Angélique: roman" 17th
"Angélique" 18th
"Angelique. 3 Autumn alone" 19th
"Angelique. 2 The long way" 20th
"Angelique. 1 Buffalo Hunt" 21st

With ElasticsearchBoostFieldMatch emabled
"Angélique" 1ST! WOW!!
"Angelique. 1 Buffalo Hunt" 2nd!!
"Angelique. 4 Angel in the snow" 3rd
"Angélique: roman" 7th
"Angelique. 3 Autumn alone" 8th
"Angelique. 2 The long way" 9th

With ElasticsearchBoostFieldMatchAmount set to 30
Same as above

I don't see a difference with ElasticsearchBoostFieldMatchAmount, I tried 10, 20, 30...
Comment 10 Caroline Cyr La Rose 2025-10-23 21:15:48 UTC
(In reply to Caroline Cyr La Rose from comment #9)
> I tried this a little bit. I used our mappings, which have weights
> author = 5.00
> subject = 10.00
> title = 20.00
> title-cover = 25.00 
> (I think the title-cover one is there because there was no
> ElasticsearchBoostFieldMatch)
> title-series = 15.00
> 
> 
> I imported 100 records which have the word "angelique" (or related word like
> angel, etc). There is, among others
> 245$a Angélique / $c Guillaume Musso 
> 245$a Angélique : $b roman / $c Guillaume Musso. 
> 245$a Angelique. $n 1 $p Buffalo Hunt $c Cora Taylor ; [map illustration,
> Sharon Matthews ; interior illustrations, James Bentley].
> 245$a Angelique. $n 3 $p Autumn alone $c Cora Taylor ; [map illustration,
> Sharon Matthews ; interior illustrations, James Bentley].
> 245$a Angelique. $n 2 $p The long way home $c Statement of responsibility,
> etc. Cora Taylor ; [map illustration, Sharon Matthews ; interior
> illustrations, James Bentley]. 
> 245$a Angelique. $n 4 $p Angel in the snow $c Cora Taylor ; [map
> illustration, Sharon Matthews ; interior illustrations, James Bentley]. 
> 
> I did these tests on KTD with ES8.I simply search for angelique in the bar
> at the top of the staff interface.
> 
> With just the weights and no prefs, the results are like this
> "Angelique. 4 Angel in the snow" 16th result
> "Angélique: roman" 17th
> "Angélique" 18th
> "Angelique. 3 Autumn alone" 19th
> "Angelique. 2 The long way" 20th
> "Angelique. 1 Buffalo Hunt" 21st
> 
> With ElasticsearchBoostFieldMatch emabled
> "Angélique" 1ST! WOW!!
> "Angelique. 1 Buffalo Hunt" 2nd!!
> "Angelique. 4 Angel in the snow" 3rd
> "Angélique: roman" 7th
> "Angelique. 3 Autumn alone" 8th
> "Angelique. 2 The long way" 9th
> 
> With ElasticsearchBoostFieldMatchAmount set to 30
> Same as above
> 
> I don't see a difference with ElasticsearchBoostFieldMatchAmount, I tried
> 10, 20, 30...

To specify, the 4th, 5th, and 6th results are not exact matches with 245$a
4th 
245$aPrésence angélique : $b messages des anges que nous rencontrons chaque jour / $c Statement of responsibility, etc. Chrissie Astell ; traduit de l'anglais par Corinne Berruel Ainsley. 

5th 
245$a La démone angélique / $c France Lorrain 

6th 
245$a La belle Angélique / $c Nadine Grelet avec la collaboration de Jacques Lamarche 


Ideally the ones with an exact match in 245$a should be at the top iiuc