When searching for book titles that may have words in many fields, e.g. 'book of hope', the boost is not enough, especially when other fields like 'title' are boosted. In production with title boost of 32 the general results are getting a score of ~600 The boost query is getting a score of ~10 - not enough to boost past the general results We can try a few things: 1 - Use the 'boost' field from 'title-cover' matching in the extra query 2 - Add a system preference (or hard code) to define how much to boost the additional query by 3 - Try using a boost query rather than a bool query: https://www.elastic.co/docs/reference/query-languages/query-dsl/query-dsl-boosting-query
Created attachment 187422 [details] [review] Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch. You can add any positive number, or a number between 0 and 1 for negative boost To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all 6 - Repeat searches, note exact titles are boosted 7 - Test with more titles as you are able
Created attachment 187425 [details] [review] Bug 40853: Unit tests
Created attachment 187500 [details] [review] Bug 40853: Add ElasticsearchBoostFieldMatchAmount system preference This adds a new system preference to control the amount of boost. The default of 0 will act as before this patch. You can add any positive number, or a number between 0 and 1 for negative boost To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all 6 - Repeat searches, note exact titles are boosted 7 - Test with more titles as you are able Signed-off-by: Jason Robb <jrobb@sekls.org>
Created attachment 187501 [details] [review] Bug 40853: Unit tests Signed-off-by: Jason Robb <jrobb@sekls.org>
I tested this with a batch of 1000 records from our production server for a search on "Book of Hope" in ktd and it worked well! My exact title search went from #649 and #651 to #1 and #2. A few notes of things I ran into: I tested with es8 first and didn't need the patch -- enabling the boost syspref with es8 worked. I tested with es7 after that (which is what our current production server is running) and the issue was present; the syspref didn't do any sort of boosting. When I applied the patch and enabled the boost and left the amount at 0 my search results were much better. I tried playing with the new amount value for the syspref and it didn't seem to do much, but the patch still gave me much better exact title boosting regardless so I went ahead and signed off.
I wanted to QA this, but I could not see any difference between before/after applying the patch, and the search result seemed correct before applying. Maybe the test plan can be improved? For example I have no clue what the expected search results should be for 'novel' / 'a novel' (because the newly created books with those titles where shown at the start of the list even before applying the patch)
Created attachment 187582 [details] MARC records for testing This marc file contains 1000 bibs with the terms "book" "of" and "hope" somewhere in the records. There are two records with the exact title "The book of hope". To test with these records: 1.) Import the records 2.) Do a keyword search for book of hope (no quotes) and note the exact titles are not near the top of the results 3.) Enable ElasticsearchBoostFieldMatch, do the same search, still not boosted 4.) Apply the patch 5.) Search again with ElasticsearchBoostFieldMatch enabled, the exact titles should be at the top of the results this time As I noted in another comment, this only happens with es7. When testing with es8 the search does get boosted at step 3.
(In reply to Thomas Klausner from comment #6) > I wanted to QA this, but I could not see any difference between before/after > applying the patch, and the search result seemed correct before applying. > > Maybe the test plan can be improved? For example I have no clue what the > expected search results should be for 'novel' / 'a novel' (because the newly > created books with those titles where shown at the start of the list even > before applying the patch) Try adding weights to fields before applying the patch as well: http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl Set weights like: title 32 author 16 subject 8
I tried this a little bit. I used our mappings, which have weights author = 5.00 subject = 10.00 title = 20.00 title-cover = 25.00 (I think the title-cover one is there because there was no ElasticsearchBoostFieldMatch) title-series = 15.00 I imported 100 records which have the word "angelique" (or related word like angel, etc). There is, among others 245$a Angélique / $c Guillaume Musso 245$a Angélique : $b roman / $c Guillaume Musso. 245$a Angelique. $n 1 $p Buffalo Hunt $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 245$a Angelique. $n 3 $p Autumn alone $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 245$a Angelique. $n 2 $p The long way home $c Statement of responsibility, etc. Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. 245$a Angelique. $n 4 $p Angel in the snow $c Cora Taylor ; [map illustration, Sharon Matthews ; interior illustrations, James Bentley]. I did these tests on KTD with ES8.I simply search for angelique in the bar at the top of the staff interface. With just the weights and no prefs, the results are like this "Angelique. 4 Angel in the snow" 16th result "Angélique: roman" 17th "Angélique" 18th "Angelique. 3 Autumn alone" 19th "Angelique. 2 The long way" 20th "Angelique. 1 Buffalo Hunt" 21st With ElasticsearchBoostFieldMatch emabled "Angélique" 1ST! WOW!! "Angelique. 1 Buffalo Hunt" 2nd!! "Angelique. 4 Angel in the snow" 3rd "Angélique: roman" 7th "Angelique. 3 Autumn alone" 8th "Angelique. 2 The long way" 9th With ElasticsearchBoostFieldMatchAmount set to 30 Same as above I don't see a difference with ElasticsearchBoostFieldMatchAmount, I tried 10, 20, 30...
(In reply to Caroline Cyr La Rose from comment #9) > I tried this a little bit. I used our mappings, which have weights > author = 5.00 > subject = 10.00 > title = 20.00 > title-cover = 25.00 > (I think the title-cover one is there because there was no > ElasticsearchBoostFieldMatch) > title-series = 15.00 > > > I imported 100 records which have the word "angelique" (or related word like > angel, etc). There is, among others > 245$a Angélique / $c Guillaume Musso > 245$a Angélique : $b roman / $c Guillaume Musso. > 245$a Angelique. $n 1 $p Buffalo Hunt $c Cora Taylor ; [map illustration, > Sharon Matthews ; interior illustrations, James Bentley]. > 245$a Angelique. $n 3 $p Autumn alone $c Cora Taylor ; [map illustration, > Sharon Matthews ; interior illustrations, James Bentley]. > 245$a Angelique. $n 2 $p The long way home $c Statement of responsibility, > etc. Cora Taylor ; [map illustration, Sharon Matthews ; interior > illustrations, James Bentley]. > 245$a Angelique. $n 4 $p Angel in the snow $c Cora Taylor ; [map > illustration, Sharon Matthews ; interior illustrations, James Bentley]. > > I did these tests on KTD with ES8.I simply search for angelique in the bar > at the top of the staff interface. > > With just the weights and no prefs, the results are like this > "Angelique. 4 Angel in the snow" 16th result > "Angélique: roman" 17th > "Angélique" 18th > "Angelique. 3 Autumn alone" 19th > "Angelique. 2 The long way" 20th > "Angelique. 1 Buffalo Hunt" 21st > > With ElasticsearchBoostFieldMatch emabled > "Angélique" 1ST! WOW!! > "Angelique. 1 Buffalo Hunt" 2nd!! > "Angelique. 4 Angel in the snow" 3rd > "Angélique: roman" 7th > "Angelique. 3 Autumn alone" 8th > "Angelique. 2 The long way" 9th > > With ElasticsearchBoostFieldMatchAmount set to 30 > Same as above > > I don't see a difference with ElasticsearchBoostFieldMatchAmount, I tried > 10, 20, 30... To specify, the 4th, 5th, and 6th results are not exact matches with 245$a 4th 245$aPrésence angélique : $b messages des anges que nous rencontrons chaque jour / $c Statement of responsibility, etc. Chrissie Astell ; traduit de l'anglais par Corinne Berruel Ainsley. 5th 245$a La démone angélique / $c France Lorrain 6th 245$a La belle Angélique / $c Nadine Grelet avec la collaboration de Jacques Lamarche Ideally the ones with an exact match in 245$a should be at the top iiuc