| Summary: | ElasticsearchBoostFieldMatch - needs to boost results more | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Nick Clemens (kidclamp) <nick> |
| Component: | Searching - Elasticsearch | Assignee: | Nick Clemens (kidclamp) <nick> |
| Status: | Signed Off --- | QA Contact: | Thomas Klausner <domm> |
| Severity: | normal | ||
| Priority: | P5 - low | CC: | ahernandez, caroline.cyr-la-rose, Chip.Halvorsen, domm, esther.melander, jrobb, michaela.sieber, mspinney |
| Version: | Main | ||
| Hardware: | All | ||
| OS: | All | ||
| GIT URL: | Initiative type: | --- | |
| Sponsorship status: | --- | Comma delimited list of Sponsors: | |
| Crowdfunding goal: | 0 | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: | Version(s) released in: | ||
| Circulation function: | |||
| Attachments: |
Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference
Bug 40853: Unit tests Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference Bug 40853: Unit tests MARC records for testing |
||
|
Description
Nick Clemens (kidclamp)
2025-09-22 18:47:54 UTC
Created attachment 187422 [details] [review] Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch. You can add any positive number, or a number between 0 and 1 for negative boost To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all 6 - Repeat searches, note exact titles are boosted 7 - Test with more titles as you are able Created attachment 187425 [details] [review] Bug 40853: Unit tests Created attachment 187500 [details] [review] Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch. You can add any positive number, or a number between 0 and 1 for negative boost To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all 6 - Repeat searches, note exact titles are boosted 7 - Test with more titles as you are able Signed-off-by: Jason Robb <jrobb@sekls.org> Created attachment 187501 [details] [review] Bug 40853: Unit tests Signed-off-by: Jason Robb <jrobb@sekls.org> I tested this with a batch of 1000 records from our production server for a search on "Book of Hope" in ktd and it worked well! My exact title search went from #649 and #651 to #1 and #2. A few notes of things I ran into: I tested with es8 first and didn't need the patch -- enabling the boost syspref with es8 worked. I tested with es7 after that (which is what our current production server is running) and the issue was present; the syspref didn't do any sort of boosting. When I applied the patch and enabled the boost and left the amount at 0 my search results were much better. I tried playing with the new amount value for the syspref and it didn't seem to do much, but the patch still gave me much better exact title boosting regardless so I went ahead and signed off. I wanted to QA this, but I could not see any difference between before/after applying the patch, and the search result seemed correct before applying. Maybe the test plan can be improved? For example I have no clue what the expected search results should be for 'novel' / 'a novel' (because the newly created books with those titles where shown at the start of the list even before applying the patch) Created attachment 187582 [details]
MARC records for testing
This marc file contains 1000 bibs with the terms "book" "of" and "hope" somewhere in the records.
There are two records with the exact title "The book of hope".
To test with these records:
1.) Import the records
2.) Do a keyword search for book of hope (no quotes) and note the exact titles are not near the top of the results
3.) Enable ElasticsearchBoostFieldMatch, do the same search, still not boosted
4.) Apply the patch
5.) Search again with ElasticsearchBoostFieldMatch enabled, the exact titles should be at the top of the results this time
As I noted in another comment, this only happens with es7. When testing with es8 the search does get boosted at step 3.
(In reply to Thomas Klausner from comment #6) > I wanted to QA this, but I could not see any difference between before/after > applying the patch, and the search result seemed correct before applying. > > Maybe the test plan can be improved? For example I have no clue what the > expected search results should be for 'novel' / 'a novel' (because the newly > created books with those titles where shown at the start of the list even > before applying the patch) Try adding weights to fields before applying the patch as well: http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl Set weights like: title 32 author 16 subject 8 I tried this a little bit. I used our mappings, which have weights author = 5.00 subject = 10.00 title = 20.00 title-cover = 25.00 (I think the title-cover one is there because there was no ElasticsearchBoostFieldMatch) title-series = 15.00 I imported 100 records which have the word "angelique" (or related word like angel, etc). There is, among others 245$a Angélique / $c Guillaume Musso 245$a Angélique : $b roman / $c Guillaume Musso. 245$a Angelique. $n 1 $p Buffalo Hunt $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 245$a Angelique. $n 3 $p Autumn alone $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 245$a Angelique. $n 2 $p The long way home $c Statement of responsibility, etc. Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 245$a Angelique. $n 4 $p Angel in the snow $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. I did these tests on KTD with ES8.I simply search for angelique in the bar at the top of the staff interface. With just the weights and no prefs, the results are like this "Angelique. 4 Angel in the snow" 16th result "Angélique: roman" 17th "Angélique" 18th "Angelique. 3 Autumn alone" 19th "Angelique. 2 The long way" 20th "Angelique. 1 Buffalo Hunt" 21st With ElasticsearchBoostFieldMatch emabled "Angélique" 1ST! WOW!! "Angelique. 1 Buffalo Hunt" 2nd!! "Angelique. 4 Angel in the snow" 3rd "Angélique: roman" 7th "Angelique. 3 Autumn alone" 8th "Angelique. 2 The long way" 9th With ElasticsearchBoostFieldMatchAmount set to 30 Same as above I don't see a difference with ElasticsearchBoostFieldMatchAmount, I tried 10, 20, 30... (In reply to Caroline Cyr La Rose from comment #9) > I tried this a little bit. I used our mappings, which have weights > author = 5.00 > subject = 10.00 > title = 20.00 > title-cover = 25.00 > (I think the title-cover one is there because there was no > ElasticsearchBoostFieldMatch) > title-series = 15.00 > > > I imported 100 records which have the word "angelique" (or related word like > angel, etc). There is, among others > 245$a Angélique / $c Guillaume Musso > 245$a Angélique : $b roman / $c Guillaume Musso. > 245$a Angelique. $n 1 $p Buffalo Hunt $c Cora Taylor ; [map illustration, > Sharon Matthews ; interior illustrations, James Bentley]. > 245$a Angelique. $n 3 $p Autumn alone $c Cora Taylor ; [map illustration, > Sharon Matthews ; interior illustrations, James Bentley]. > 245$a Angelique. $n 2 $p The long way home $c Statement of responsibility, > etc. Cora Taylor ; [map illustration, Sharon Matthews ; interior > illustrations, James Bentley]. > 245$a Angelique. $n 4 $p Angel in the snow $c Cora Taylor ; [map > illustration, Sharon Matthews ; interior illustrations, James Bentley]. > > I did these tests on KTD with ES8.I simply search for angelique in the bar > at the top of the staff interface. > > With just the weights and no prefs, the results are like this > "Angelique. 4 Angel in the snow" 16th result > "Angélique: roman" 17th > "Angélique" 18th > "Angelique. 3 Autumn alone" 19th > "Angelique. 2 The long way" 20th > "Angelique. 1 Buffalo Hunt" 21st > > With ElasticsearchBoostFieldMatch emabled > "Angélique" 1ST! WOW!! > "Angelique. 1 Buffalo Hunt" 2nd!! > "Angelique. 4 Angel in the snow" 3rd > "Angélique: roman" 7th > "Angelique. 3 Autumn alone" 8th > "Angelique. 2 The long way" 9th > > With ElasticsearchBoostFieldMatchAmount set to 30 > Same as above > > I don't see a difference with ElasticsearchBoostFieldMatchAmount, I tried > 10, 20, 30... To specify, the 4th, 5th, and 6th results are not exact matches with 245$a 4th 245$aPrésence angélique : $b messages des anges que nous rencontrons chaque jour / $c Statement of responsibility, etc. Chrissie Astell ; traduit de l'anglais par Corinne Berruel Ainsley. 5th 245$a La démone angélique / $c France Lorrain 6th 245$a La belle Angélique / $c Nadine Grelet avec la collaboration de Jacques Lamarche Ideally the ones with an exact match in 245$a should be at the top iiuc |