Bug 24155 - Weights should be (optionally) applied to Advanced search
Summary: Weights should be (optionally) applied to Advanced search
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Nick Clemens (kidclamp)
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-12-03 18:14 UTC by Nick Clemens (kidclamp)
Modified: 2021-06-14 21:33 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This patch adds the weighting of search results to searches made via the 'Advanced search' interface. Weights, defined in Administration section, boost ranking of results when specified fields are matched in a search query. The weights will not affect index-specific queries, but are useful for keyword or queries with limits applied and so should be applied unless the user specifies not to.
Version(s) released in:
20.11.00
Circulation function:

Attachments
Bug 24155: Make wieghting an option on advanced search (6.01 KB, patch)
2020-02-26 15:12 UTC, Nick Clemens (kidclamp) 		Details | Diff | Splinter Review
Bug 24155: Make wieghting an option on advanced search (6.84 KB, patch)
2020-02-26 16:57 UTC, Nick Clemens (kidclamp) 		Details | Diff | Splinter Review
Bug 24155: Make wieghting an option on advanced search (6.87 KB, patch)
2020-03-18 14:28 UTC, Nick Clemens (kidclamp) 		Details | Diff | Splinter Review
Bug 24155: Make wieghting an option on advanced search (6.94 KB, patch)
2020-05-06 16:05 UTC, Myka Kennedy Stephens 		Details | Diff | Splinter Review
Bug 24155: Apply wieghts if not advanced search (1.50 KB, patch)
2020-05-18 17:33 UTC, Nick Clemens (kidclamp) 		Details | Diff | Splinter Review
Bug 24155: Make wieghting an option on advanced search (7.01 KB, patch)
2020-05-18 18:16 UTC, Andrew Fuerste-Henry 		Details | Diff | Splinter Review
Bug 24155: Apply wieghts if not advanced search (1.56 KB, patch)
2020-05-18 18:17 UTC, Andrew Fuerste-Henry 		Details | Diff | Splinter Review
Bug 24155: Make wieghting an option on advanced search (7.06 KB, patch)
2020-05-18 18:27 UTC, Andrew Fuerste-Henry 		Details | Diff | Splinter Review
Bug 24155: Apply wieghts if not advanced search (1.61 KB, patch)
2020-05-18 18:27 UTC, Andrew Fuerste-Henry 		Details | Diff | Splinter Review
OPAC advanced search with weight search option (89.23 KB, image/png)
2020-08-27 22:14 UTC, Katrin Fischer 		Details
Bug 24155: (follow-up) Add span with id to wieghted search option (2.52 KB, patch)
2020-08-31 12:38 UTC, Nick Clemens (kidclamp) 		Details | Diff | Splinter Review
Bug 24155: (follow-up) Template fixes and updates (4.86 KB, patch)
2020-09-18 15:49 UTC, Owen Leonard 		Details | Diff | Splinter Review
Bug 24155: Make wieghting an option on advanced search (7.09 KB, patch)
2020-09-18 15:51 UTC, Owen Leonard 		Details | Diff | Splinter Review
Bug 24155: Apply wieghts if not advanced search (1.61 KB, patch)
2020-09-18 15:51 UTC, Owen Leonard 		Details | Diff | Splinter Review
Bug 24155: (follow-up) Add span with id to wieghted search option (2.54 KB, patch)
2020-09-18 15:52 UTC, Owen Leonard 		Details | Diff | Splinter Review
Bug 24155: (follow-up) Template fixes and updates (4.86 KB, patch)
2020-09-18 15:52 UTC, Owen Leonard 		Details | Diff | Splinter Review
Bug 24155: Make weighting an option on advanced search (7.21 KB, patch)
2020-09-20 17:59 UTC, Katrin Fischer 		Details | Diff | Splinter Review
Bug 24155: Apply weights if not advanced search (1.72 KB, patch)
2020-09-20 17:59 UTC, Katrin Fischer 		Details | Diff | Splinter Review
Bug 24155: (follow-up) Add span with id to weighted search option (2.65 KB, patch)
2020-09-20 17:59 UTC, Katrin Fischer 		Details | Diff | Splinter Review
Bug 24155: (follow-up) Template fixes and updates (4.97 KB, patch)
2020-09-20 17:59 UTC, Katrin Fischer 		Details | Diff | Splinter Review
Bug 24155: Fix parameter order to fix search_utf8.t (1.80 KB, patch)
2020-10-01 13:06 UTC, Jonathan Druart 		Details | Diff | Splinter Review
Show Obsolete (15) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2019-12-03 18:14:56 UTC 
When searching Keywords via the advanced search we should apply the weights to indexes. This would allow users to use the limits on the advanced search and still get more relevant results
Comment 1 AspenCat Team 2019-12-04 15:56:12 UTC 
This would be very helpful for our libraries that are accustomed to using the advanced search. - Bob Bennhoff
Comment 2 Nick Clemens (kidclamp) 2020-02-26 15:12:51 UTC Comment hidden (obsolete)
Comment 3 Nick Clemens (kidclamp) 2020-02-26 16:57:50 UTC Comment hidden (obsolete)
Comment 4 Myka Kennedy Stephens 2020-03-03 00:51:17 UTC 
Patch failed to apply:

Applying: Bug 24155: Make wieghting an option on advanced search
Using index info to reconstruct a base tree...
M       cataloguing/addbooks.pl
M       koha-tmpl/opac-tmpl/bootstrap/en/includes/masthead.inc
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/opac-tmpl/bootstrap/en/includes/masthead.inc
Auto-merging cataloguing/addbooks.pl
CONFLICT (content): Merge conflict in cataloguing/addbooks.pl
error: Failed to merge in the changes.
Patch failed at 0001 Bug 24155: Make wieghting an option on advanced search
Comment 5 Myka Kennedy Stephens 2020-03-03 01:06:28 UTC 
I would also like to add that as I was doing my base-line testing for this bug, I found something interesting. When clicking on a record in the result list, then clicking on the button to return to the results, the search re-executes and applies weights. If a user had performed a search using advanced search where weights are not applied, then used the button to return to the results list, they would find that the results were in a different order.

The work-around is to just use the browser's back button to return to the advanced search results list. If we're giving people the option to apply weights to the advanced search, however, this inconsistency will continue to happen unless there's a way to specify the return to results link to go back to the previous results instead of launching a new search.

I did notice that there is a search_id value in the URL when the search re-executes that is not there in the URL for the advanced search:

http://localhost:8081/cgi-bin/koha/catalogue/search.pl?advsearch=1&idx=kw&q=computer&op=and&idx=kw&q=program*&op=and&idx=kw&sort_by=relevance
vs.
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?idx=kw&q=computer&op=and&idx=kw&q=program*&op=and&idx=kw&searchid=scs_1583197470738&offset=0
Comment 6 Nick Clemens (kidclamp) 2020-03-18 14:28:05 UTC Comment hidden (obsolete)
Comment 7 Myka Kennedy Stephens 2020-05-06 16:05:54 UTC Comment hidden (obsolete)
Comment 8 Myka Kennedy Stephens 2020-05-06 16:09:02 UTC 
Worked as expected. Searches with and without weights applied returned expected results. The behavior I was observing before applying a patch where looking at a result in detail and then returning to the results list to find the sorting had changed was also fixed. Now, if you choose not to apply weights, returning to the search results takes you back to the un-weighted results.

Thank you! Nice work!
Comment 9 Andrew Fuerste-Henry 2020-05-18 16:33:00 UTC 
With this patch, weights don't seem to be applied in a basic search.

To recreate (in the default data for master):
1 - verify you're using ES and have no weights set
2 - do a basic search for "music"
3 - Note your top three results are "New Musical Express," "Miles Davis," and "All Music Guide to Soul"
4 - Enter 100 in weights for title index
5 - Reload your search
6 - Note your top three results are now "New Musical Express," "All Music Guide to Soul," and "Music Theory for Dummies."
7 - Apply patch
8 - Reload search and see you're back to your results ordering from step 3
9 - go to advanced search, perform a keyword search for "music" with "Apply field weightings to search" box checked
10 - Note these results match what's in step 6
Comment 10 Nick Clemens (kidclamp) 2020-05-18 17:33:25 UTC 
Created attachment 105033 [details] [review]
Bug 24155: Apply wieghts if not advanced search
Comment 11 Andrew Fuerste-Henry 2020-05-18 18:16:56 UTC Comment hidden (obsolete)
Comment 12 Andrew Fuerste-Henry 2020-05-18 18:17:00 UTC Comment hidden (obsolete)
Comment 13 Andrew Fuerste-Henry 2020-05-18 18:27:38 UTC 
Created attachment 105041 [details] [review]
Bug 24155: Make wieghting an option on advanced search

Currently we apply weighting to all searches except advanced search. The theory
being that when selecting indexes we don't want to apply weights. When searching
in ES weights are only applied to relevant results so it doesn't matter.

i.e. if weighting author*100 but searching subject, a term matching the subject search in author
is not boosted.

Given this, we should always apply weights, unless the user wishes not to

To test:
1 - Set some weighting
2 - Do some searches
3 - Note your terms and results, try advanced and regular searches specifying indexes or not
4 - Apply patch
5 - Note that opac and staff advanced search have option to apply weights
6 - Compare searches after the patch to see how weighting affects, it should be beneficial or not at all

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Comment 14 Andrew Fuerste-Henry 2020-05-18 18:27:42 UTC 
Created attachment 105042 [details] [review]
Bug 24155: Apply wieghts if not advanced search

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Comment 15 Katrin Fischer 2020-08-27 22:13:34 UTC 
Some first thoughts here:

I feel the GUI and maybe wording needs a little work.  

1) the checkbox option appears kind of out of context on top of the form. I wonder if it would be better to place it closer to the search button - but that's problematic as we have 2 of those. I have no really good idea yet, but wanted to to get it out there. Attaching a screenshot.

2) The wording is currently: I wonder if "Apply field weightings to search"
I wonder if this works for patrons or if there is a better/shorter description we could use.
Comment 16 Katrin Fischer 2020-08-27 22:14:01 UTC 
Created attachment 109264 [details]
OPAC advanced search with weight search option
Comment 17 Katrin Fischer 2020-08-27 22:17:39 UTC 
About 1) Staff interface: what about putting it into the floating toolbar?
Comment 18 Katrin Fischer 2020-08-27 22:19:23 UTC 
Do you have some suggestions for a good weighting configuration to test with?
Comment 19 Katrin Fischer 2020-08-27 22:21:42 UTC 
When I try to add weights, I have:

 An error occurred when updating Elasticsearch index mappings: Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Can't parse [index] value [] for field [copynumber__sort], expected [true] or [false]". Index needs to be recreated and reindexed.
An error occurred when updating Elasticsearch index mappings: Unable to update mappings for index "koha_kohadev_authorities". Reason was: "Can't parse [index] value [] for field [personal-name-heading__sort], expected [true] or [false]". Index needs to be recreated and reindexed.
Index '' needs to be recreated.
Index '' needs to be recreated. 

I've tried resetting the mappings, but same errors.
Comment 20 Nick Clemens (kidclamp) 2020-08-28 10:20:09 UTC 
(In reply to Katrin Fischer from comment #18)
> Do you have some suggestions for a good weighting configuration to test with?

Our standard for partners is:
title: 32
author: 16
subject: 8
title-series: 4
contents: 2
Comment 21 Nick Clemens (kidclamp) 2020-08-28 10:23:06 UTC 
(In reply to Katrin Fischer from comment #19)
> When I try to add weights, I have:
> 
>  An error occurred when updating Elasticsearch index mappings: Unable to
> update mappings for index "koha_kohadev_biblios". Reason was: "Can't parse
> [index] value [] for field [copynumber__sort], expected [true] or [false]".
> Index needs to be recreated and reindexed.
> An error occurred when updating Elasticsearch index mappings: Unable to
> update mappings for index "koha_kohadev_authorities". Reason was: "Can't
> parse [index] value [] for field [personal-name-heading__sort], expected
> [true] or [false]". Index needs to be recreated and reindexed.
> Index '' needs to be recreated.
> Index '' needs to be recreated. 
> 
> I've tried resetting the mappings, but same errors.

I do not recreate, it sounds like your indexes aren't there, have you indexed in ES?
Comment 22 Katrin Fischer 2020-08-28 10:25:07 UTC 
(In reply to Nick Clemens from comment #21)
> (In reply to Katrin Fischer from comment #19)
> > When I try to add weights, I have:
> > 
> >  An error occurred when updating Elasticsearch index mappings: Unable to
> > update mappings for index "koha_kohadev_biblios". Reason was: "Can't parse
> > [index] value [] for field [copynumber__sort], expected [true] or [false]".
> > Index needs to be recreated and reindexed.
> > An error occurred when updating Elasticsearch index mappings: Unable to
> > update mappings for index "koha_kohadev_authorities". Reason was: "Can't
> > parse [index] value [] for field [personal-name-heading__sort], expected
> > [true] or [false]". Index needs to be recreated and reindexed.
> > Index '' needs to be recreated.
> > Index '' needs to be recreated. 
> > 
> > I've tried resetting the mappings, but same errors.
> 
> I do not recreate, it sounds like your indexes aren't there, have you
> indexed in ES?

Yes - search was working.
Comment 23 Katrin Fischer 2020-08-29 11:24:07 UTC 
Ok, giving this another go:
- Testing on kohadevbox
- reset_all - nice and fresh database
- kshell
- Reindexing: ./misc/search_tools/rebuild_elasticsearch.pl -v -r
- Preference SearchEngine set to Elasticsearch
- Searching in staff - successful
- Administration > Elasticsearch
- Trying to set the suggested weight configuration, starting with: title: 32
  On first tab, only entered the number and then hit Save.

 An error occurred when updating Elasticsearch index mappings: Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Can't parse [index] value [] for field [date-of-publication__sort], expected [true] or [false]". Index needs to be recreated and reindexed.
An error occurred when updating Elasticsearch index mappings: Unable to update mappings for index "koha_kohadev_authorities". Reason was: "Can't parse [index] value [] for field [match-heading-see-from__sort], expected [true] or [false]". Index needs to be recreated and reindexed.
Index '' needs to be recreated.
Index '' needs to be recreated. 

:(
Comment 24 Katrin Fischer 2020-08-29 11:24:57 UTC 
I can't fail this as it's not related to the patch set, but I also can't continue testing.
Comment 25 Nick Clemens (kidclamp) 2020-08-31 12:38:37 UTC Comment hidden (obsolete)
Comment 26 Owen Leonard 2020-09-18 15:49:29 UTC Comment hidden (obsolete)
Comment 27 Owen Leonard 2020-09-18 15:51:51 UTC 
Created attachment 110406 [details] [review]
Bug 24155: Make wieghting an option on advanced search

Currently we apply weighting to all searches except advanced search. The theory
being that when selecting indexes we don't want to apply weights. When searching
in ES weights are only applied to relevant results so it doesn't matter.

i.e. if weighting author*100 but searching subject, a term matching the subject search in author
is not boosted.

Given this, we should always apply weights, unless the user wishes not to

To test:
1 - Set some weighting
2 - Do some searches
3 - Note your terms and results, try advanced and regular searches specifying indexes or not
4 - Apply patch
5 - Note that opac and staff advanced search have option to apply weights
6 - Compare searches after the patch to see how weighting affects, it should be beneficial or not at all

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Comment 28 Owen Leonard 2020-09-18 15:51:57 UTC 
Created attachment 110407 [details] [review]
Bug 24155: Apply wieghts if not advanced search

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Comment 29 Owen Leonard 2020-09-18 15:52:02 UTC 
Created attachment 110408 [details] [review]
Bug 24155: (follow-up) Add span with id to wieghted search option
Comment 30 Owen Leonard 2020-09-18 15:52:06 UTC 
Created attachment 110409 [details] [review]
Bug 24155: (follow-up) Template fixes and updates

This patch makes a few minor changes to the OPAC and staff interface
templates:

- In both, the "Apply field weights" checkbox will only appear after the
  "More options" button has been clicked. Otherwise a hidden input sets
  the value to 1.
- In both, the checkbox now appears below the search fields.
- Checkboxes are now wrapped in the <label> tag to ensure the label is
  clickable.
Comment 31 Katrin Fischer 2020-09-20 17:59:05 UTC 
Created attachment 110431 [details] [review]
Bug 24155: Make weighting an option on advanced search

Currently we apply weighting to all searches except advanced search. The theory
being that when selecting indexes we don't want to apply weights. When searching
in ES weights are only applied to relevant results so it doesn't matter.

i.e. if weighting author*100 but searching subject, a term matching the subject search in author
is not boosted.

Given this, we should always apply weights, unless the user wishes not to

To test:
1 - Set some weighting
2 - Do some searches
3 - Note your terms and results, try advanced and regular searches specifying indexes or not
4 - Apply patch
5 - Note that opac and staff advanced search have option to apply weights
6 - Compare searches after the patch to see how weighting affects, it should be beneficial or not at all

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 32 Katrin Fischer 2020-09-20 17:59:12 UTC 
Created attachment 110432 [details] [review]
Bug 24155: Apply weights if not advanced search

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 33 Katrin Fischer 2020-09-20 17:59:18 UTC 
Created attachment 110433 [details] [review]
Bug 24155: (follow-up) Add span with id to weighted search option

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 34 Katrin Fischer 2020-09-20 17:59:26 UTC 
Created attachment 110434 [details] [review]
Bug 24155: (follow-up) Template fixes and updates

This patch makes a few minor changes to the OPAC and staff interface
templates:

- In both, the "Apply field weights" checkbox will only appear after the
  "More options" button has been clicked. Otherwise a hidden input sets
  the value to 1.
- In both, the checkbox now appears below the search fields.
- Checkboxes are now wrapped in the <label> tag to ensure the label is
  clickable.

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 35 Jonathan Druart 2020-10-01 09:44:26 UTC 
Pushed to master for 20.11, thanks to everybody involved!
Comment 36 Jonathan Druart 2020-10-01 12:36:54 UTC 
We have a failure, can you have a look please?

t/db_dependent/www/search_utf8.t .. 24/99 
#   Failed test 'Base is like "(?^u:opac-search.pl\?(idx=&)?q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)"'
#   at t/db_dependent/www/search_utf8.t line 300.
#          got: "http://koha:8080/cgi-bin/koha/opac-search.pl?weigh"...
#       length: 93
#     doesn't match '(?^u:opac-search.pl\?(idx=&)?q=%CE%91%CE%B8%CE%AE%CE%BD%CE%B1)'
t/db_dependent/www/search_utf8.t .. 57/99 
#   Failed test 'Base is like "(?^u:opac-search.pl\?(idx=&)?q=Ram%C3%B2n)"'
#   at t/db_dependent/www/search_utf8.t line 300.
#          got: "http://koha:8080/cgi-bin/koha/opac-search.pl?weigh"...
#       length: 73
#     doesn't match '(?^u:opac-search.pl\?(idx=&)?q=Ram%C3%B2n)'
t/db_dependent/www/search_utf8.t .. 89/99 
#   Failed test 'Base is like "(?^u:opac-search.pl\?(idx=&)?q=%F0%A0%BB%BAtomasito%F0%A0%BB%BA)"'
#   at t/db_dependent/www/search_utf8.t line 300.
#          got: "http://koha:8080/cgi-bin/koha/opac-search.pl?weigh"...
#       length: 95
#     doesn't match '(?^u:opac-search.pl\?(idx=&)?q=%F0%A0%BB%BAtomasito%F0%A0%BB%BA)
Comment 37 Jonathan Druart 2020-10-01 13:06:48 UTC 
Created attachment 111039 [details] [review]
Bug 24155: Fix parameter order to fix search_utf8.t
Comment 38 Lucas Gass (lukeg) 2020-10-20 17:55:02 UTC 
enhancement will not be backported to 20.05.x
Comment 39 Lucas Gass (lukeg) 2020-10-20 18:11:52 UTC 
enhancement will not be backported to 20.05.x