When searching in Zebra, the backed actually did some combined searches, one of them being an exact title search. When libraries switch to Elastic, they notice exact titles aren't boosted in the same way. So a search for 'The Help', 'It', or 'To die for' won't necessarily return the expected result first. We can use the 'bool' query in Elastic to add a more exact search for title
Created attachment 175423 [details] [review] Bug 38694: Title boost ES (WIP) This patch is a rough example, needs work and tests
Personally, I've always disliked the query manipulation we did in Zebra. A few of my libraries have wanted more exact titles to float to the top of the results, and I've found field weighting the title index has been sufficient for my expectations and theirs. Can you explain more about what this patch does?
(In reply to David Cook from comment #2) > A few of my libraries have wanted more exact titles to float to the top of > the results, and I've found field weighting the title index has been > sufficient for my expectations and theirs. Exactly, I have the same observations--according to my tests and in real world installations field weighting makes a good job in this regard.
I had asked about a way to do exact title searches in the Elasticsearch channel a while ago. We have a lot of special libraries where searching for a "one word title" can give a lot of results if it's some common word. "Nature" comes to mind as a popular serial example. They are currently able to do exact search with Zebra, so it's something we have missed in Elasticsearch so far. Can you tell if field boosting is enough here to make it first result?
I would try putting weight 10 for title-no-punctuation and either disable QueryAutoTruncate of put search string in quotes. Are than the search results satisfactory or still not...?
(In reply to Janusz Kaczmarek from comment #5) > I would try putting weight 10 for title-no-punctuation and either disable > QueryAutoTruncate of put search string in quotes. Are than the search > results satisfactory or still not...? (NB no reindex is necessary after changing the weights.)
(In reply to Janusz Kaczmarek from comment #5) > I would try putting weight 10 for title-no-punctuation and either disable > QueryAutoTruncate of put search string in quotes. Are than the search > results satisfactory or still not...? We don't have a test system with data yet, so I cannot test it at the moment. But I will keep it in mind.
The problem comes especially when other titles include the full title that is being searched for. This patch adds a second query sent to ES along with the user's original query. The queries are sent as a 'bool' compound search, such that the original query is a 'must' - so we will only return results that are in the initial query. The query the patches add is a 'should' query. This simply serves as an additional boost - so where the title field matches the terms entered exactly we will boost that result to the top. To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all 6 - Repeat searches, note exact titles are boosted It's not going to affect relevancy outside of exact title matches, though I will follow up with another patch soon - I think we can apply the same logic to other indexes - i.e. if I am searching for 'title' or 'kw' index - then add 'title-cover' but if I am searching 'subject' then add an field match for 'subject' so that "Underwater basket weaving" would match an exact subject over a record with "Underwater sealife" "Basket fish" "Weaving fish" (I know, dumb example) As for the why of this: Example from a partner catalog, searching for 'To Die For' - expecting the David Baldacci book with exact title I get: A diet to die for Antiques to die for U2, Rattle and Hum (track titles contain the three words) Upstairs Downstairs ....many more results To Die For by David Baldacci I tried adding a boost of 128 (exponential above others) to title-cover - and that can get titles up to 4 or 5 in the results lists, but before 'The Help' we still get: How to help the earth Help for the haunted Help! I'm a prisoner You can also improve things by searching with quotes, or searching 'title-cover' directly, but none of this restores the ease of title searching that users experienced in Zebra. For libraries that switch this is really seen as a loss. The time I have spent explaining to libraries why this isn't a problem would be better spent adding the feature they want ;-) I am happy to make this optional, but I do think it is a necessary feature based on the amount of requests I have received.
Created attachment 175459 [details] [review] Bug 38694: Title boost ES (WIP) This patch is a rough example, will need tests and maybe an on/off switch What this patch does is: 1 - Wraps the existing search code in a a "bool" compound query as a "must". This should not affect relevancy or results of the existing searches. 2 - Before we clean/truncate terms, loop through the passed in search terms and indexes to build a new 'should' query, using the 'match' on the specified index/field that is added to the 'bool' query from above. This means that if a result from the original query is also returned here, that item will be boosted in the result. For searches on 'keyword' or 'title', or if no index is set, we use 'title-cover' as the most narrow form of title This query isn't going to help when users enter CCL (i.e. ti:To die for) and it won't boost titles from 505, series, etc when doing a general search. Nor will it have a detrimental effect, it will only boost field matches To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all 6 - Repeat searches, note exact titles are boosted
We have several libraries that would like exact title search matches to return higher in the search results.
(In reply to Nick Clemens (kidclamp) from comment #8) > You can also improve things by searching with quotes, or searching > 'title-cover' directly, but none of this restores the ease of title > searching that users experienced in Zebra. For libraries that switch this is > really seen as a loss. Admittedly I have had this conversation with a few libraries for sure. > The time I have spent explaining to libraries why this isn't a problem would > be better spent adding the feature they want ;-) Also relatable haha. > I am happy to make this optional, but I do think it is a necessary feature > based on the amount of requests I have received. That's for explaining more about the background and the patch. That helps a lot. Yeah, I think making it optional is a good idea. Defaulting to off for existing installs and maybe even defaulting to on for new installs. I'll add this to my list of things to review, as it does sound like it could be valuable. Most of my libraries are on ES now, but I have a few on Zebra still, and when they switch over... I suspect I'll get more of these queries again.
Created attachment 175840 [details] [review] Bug 38694: Add ESBoostFieldMatch option to Elasticsearch What this patch does is: 1 - Wraps the existing search code in a a "bool" compound query as a "must". This should not affect relevancy or results of the existing searches. 2 - Before we clean/truncate terms, loop through the passed in search terms and indexes to build a new 'should' query, using the 'match' on the specified index/field that is added to the 'bool' query from above. This means that if a result from the original query is also returned here, that item will be boosted in the result. For searches on 'keyword' or 'title', or if no index is set, we use 'title-cover' as the most narrow form of title This query isn't going to help when users enter CCL (i.e. ti:To die for) and it won't boost titles from 505, series, etc when doing a general search. Nor will it have a detrimental effect, it will only boost field matches To test: 1 - Add a record with 245 $a novel 2 - Add a record with 245 $a A novel : $b about things / $c by me 3 - Search for: novel 4 - Search for: a novel 5 - Apply patch, restart all, enable ESBoostFieldMatch option 6 - Repeat searches, note exact titles are boosted 7 - Experiment with other searches, turniung pref on and off to verify relevant titles are boosted when enabled 8 - Search results when disabled should not change
Created attachment 175841 [details] [review] Bug 38694: Unit tests Adjust existing and add new
Created attachment 175842 [details] [review] Bug 38694: Tidy routines for QA
(In reply to Nick Clemens (kidclamp) from comment #14) > Created attachment 175842 [details] [review] [review] > Bug 38694: Tidy routines for QA With the current work on bug 38664 I don't think we should continue to provide such patches.
(In reply to Jonathan Druart from comment #15) > (In reply to Nick Clemens (kidclamp) from comment #14) > > Created attachment 175842 [details] [review] [review] [review] > > Bug 38694: Tidy routines for QA > > With the current work on bug 38664 I don't think we should continue to > provide such patches. Wouldn't it make sense to provide such patches until after that work is done?
(In reply to David Cook from comment #16) > (In reply to Jonathan Druart from comment #15) > > (In reply to Nick Clemens (kidclamp) from comment #14) > > > Created attachment 175842 [details] [review] [review] [review] [review] > > > Bug 38694: Tidy routines for QA > > > > With the current work on bug 38664 I don't think we should continue to > > provide such patches. > > Wouldn't it make sense to provide such patches until after that work is done? I think I see Joubu's point. As we plan to do the clean-up this cycle. Do you see an advantage of continuing?
(In reply to Katrin Fischer from comment #17) > I think I see Joubu's point. As we plan to do the clean-up this cycle. Do > you see an advantage of continuing? Until the existing code is tidied, it's going to make patches harder to review. I think that's part of why we have separate tidy patches. I guess it all comes down to timing. In theory, if we do the tidy in the main patch, and the existing codebase gets tidied, that would probably yield fewer conflicts. Perhaps hypothetical fewer conflicts would be worth having a harder time reviewing for now...
(In reply to David Cook from comment #18) > (In reply to Katrin Fischer from comment #17) > > I think I see Joubu's point. As we plan to do the clean-up this cycle. Do > > you see an advantage of continuing? > > Until the existing code is tidied, it's going to make patches harder to > review. I think that's part of why we have separate tidy patches. > > I guess it all comes down to timing. In theory, if we do the tidy in the > main patch, and the existing codebase gets tidied, that would probably yield > fewer conflicts. > > Perhaps hypothetical fewer conflicts would be worth having a harder time > reviewing for now... Not the place to discuss it but quick reply anyway: I have advised already (several times) that I will provide a script to easy rebases and that you should not worry about that. The tidy patches hitting main before the global tidy will generate conflicts however, as the script won't be ready... Anyway not a big deal, just ignore this if you disagree, and hopefully everything will be tidy soon ;)
(In reply to Jonathan Druart from comment #19) > (In reply to David Cook from comment #18) > > (In reply to Katrin Fischer from comment #17) > > > I think I see Joubu's point. As we plan to do the clean-up this cycle. Do > > > you see an advantage of continuing? > > > > Until the existing code is tidied, it's going to make patches harder to > > review. I think that's part of why we have separate tidy patches. > > > > I guess it all comes down to timing. In theory, if we do the tidy in the > > main patch, and the existing codebase gets tidied, that would probably yield > > fewer conflicts. > > > > Perhaps hypothetical fewer conflicts would be worth having a harder time > > reviewing for now... > > Not the place to discuss it but quick reply anyway: I have advised already > (several times) that I will provide a script to easy rebases and that you > should not worry about that. The tidy patches hitting main before the global > tidy will generate conflicts however, as the script won't be ready... Anyway > not a big deal, just ignore this if you disagree, and hopefully everything > will be tidy soon ;) I am happy not to tidy anymore ;-) I think too, separate patches mean the devs can provide (to make QA scripts green) and RM can ignore (to make Joubu green)
> I am happy not to tidy anymore ;-) I think too, separate patches mean the > devs can provide (to make QA scripts green) and RM can ignore (to make Joubu > green) :D