Bug 20589 - Add field boosting and use elastic query fields parameter instead of deprecated _all
Summary: Add field boosting and use elastic query fields parameter instead of deprecat...
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: David Gustafsson
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-16 15:11 UTC by David Gustafsson
Modified: 2018-04-19 14:29 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 20589: Add field boosting and use query_string fields parameter (27.31 KB, patch)
2018-04-16 15:39 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 20589: Add field boosting and use query_string fields parameter (27.31 KB, patch)
2018-04-16 16:02 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 20589: Add field boosting and use query_string fields parameter (27.53 KB, patch)
2018-04-17 07:38 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 20589: Add field boosting and use query_string fields parameter (31.18 KB, patch)
2018-04-17 18:07 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 20589: Add field boosting and use query_string fields parameter (31.15 KB, patch)
2018-04-18 09:19 UTC, David Gustafsson
Details | Diff | Splinter Review
Bug 20589: Add field boosting and use query_string fields parameter (31.71 KB, patch)
2018-04-18 09:24 UTC, David Gustafsson
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Gustafsson 2018-04-16 15:11:11 UTC
Generate a list of fields for the query_string query fields parameter, with possible boosts, instead of using "_all"-field. Also add "search" flag in search_marc_to_field table so that certain mappings can be excluded from searches. Add option to include/exclude fields in query_string "fields" parameter depending on searching in OPAC or staff client. Refactor code to remove all other dependencies on "_all"-field.

(the atomicupdate sql-file is a bit obfuscated since had to be able to run in multiple time without error because of our local deploy-script, when moving this all procedure related code can be removed, only the "ALTER TABLE.."-lines are relevant.)
Comment 1 David Gustafsson 2018-04-16 15:39:03 UTC
Created attachment 74245 [details] [review]
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
   biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Comment 2 David Gustafsson 2018-04-16 16:02:38 UTC
Created attachment 74249 [details] [review]
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
   biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Comment 3 David Gustafsson 2018-04-17 07:38:38 UTC
Created attachment 74301 [details] [review]
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
   biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Comment 4 David Gustafsson 2018-04-17 07:40:29 UTC
Added a *.perl update script instead of the custom SQL-script.
Comment 5 David Gustafsson 2018-04-17 18:07:02 UTC
Created attachment 74373 [details] [review]
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
   biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Comment 6 David Gustafsson 2018-04-17 18:11:46 UTC
Had forgotten to take new "search" field into account for mappings, and also fixed where had not removed all of the _all-field mapping (which would have caused indexing to crash).
Comment 7 David Gustafsson 2018-04-18 09:19:00 UTC
Created attachment 74418 [details] [review]
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
   biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Comment 8 David Gustafsson 2018-04-18 09:23:24 UTC
Discovered search where missing in '+as', and '+select' clause, which causes indexing to crash, will fix.
Comment 9 David Gustafsson 2018-04-18 09:24:01 UTC
Created attachment 74419 [details] [review]
Bug 20589: Add field boosting and use query_string fields parameter

Generate a list of fields for the query_string query fields parameter,
with possible boosts, instead of using "_all"-field. Also add "search"
flag in search_marc_to_field table so that certain mappings can be
excluded from searches. Add option to include/exclude fields in
query_string "fields" parameter depending on searching in OPAC or staff
client. Refactor code to remove all other dependencies on "_all"-field.

How to test:
1) No reindex is really required, but still, reindex authorities and
   biblios to verify that this still works.
2) Search biblios and try to verify that this works as expected.
3) Search authorities and try to verify that this works as expected.
4) Go to "Search engine configuration"
5) Change some "Boost", "Staff client", and "OPAC" settings and save.
6) Verify that those settings where saved accordingly.
7) Click the "Biblios" or "Authorities" tab and change one or more
   "Searchable" settings
8) Verfiy that those settings where saved accordingly.
9) Try to verify that these settings has taken effect by peforming
   some biblios and/or authorities searches.

Sponsorded-by: Gothenburg Univesity Library
Comment 10 Alex Arnaud 2018-04-19 14:29:26 UTC
Searchable parameter on search fields (staff/opac) is nice.
But, is searchable parameter on mapping useful? We can just remove the mapping, right?

This patch could be rebased on bug 18316, that brings boost for simple search.