Bug 24555 - POC Elasticsearch - Use boolean queries instead of full text queries
Summary: POC Elasticsearch - Use boolean queries instead of full text queries
Status: Patch doesn't apply
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Alex Arnaud
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-01-31 16:01 UTC by Alex Arnaud
Modified: 2023-12-31 13:56 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 24555: Replace query string with boolean query (6.41 KB, patch)
2020-01-31 16:04 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 24555: add a button that shows ES query in search result (3.29 KB, patch)
2020-01-31 16:04 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 24555: Replace query string with boolean query (6.47 KB, patch)
2020-01-31 20:32 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 24555: add a button that shows ES query in search result (3.35 KB, patch)
2020-01-31 20:32 UTC, Michal Denar
Details | Diff | Splinter Review
Bug 24555: Use 'cross_fields' type for multi_match queries (1.59 KB, patch)
2020-05-06 13:24 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24555: Use 'cross_fields' type for multi_match queries (1.59 KB, patch)
2020-05-06 13:50 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24555: Remove 'mc-' prefix in facet field name (866 bytes, patch)
2020-05-19 08:24 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24555: Use operator "and" for match query (918 bytes, patch)
2020-06-01 11:23 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24555: Fix search limits (2.13 KB, patch)
2020-06-01 11:23 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24555: Fix search limits (2.18 KB, patch)
2020-06-01 12:04 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24555: Fix searches like 'field-name:term' (1.03 KB, patch)
2020-06-11 12:37 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alex Arnaud 2020-01-31 16:01:33 UTC
Here is a proof of concept for replacing current use of full text queries (with query_string) by boolean queries on biblio searches.

Advantage:
    => No search craches with special character ( "!", ")" etc...)
    => Separate query context and filter context:
        - Search scores are not altered by facets,
        - Ability to use range filter  (i.e for publication date facet)

Also, this POC adds a button "ES query" in the search results header so you can see the query sent to ES.

Code is not beautiful, there is no unit test, it's a POC.
Comment 1 Alex Arnaud 2020-01-31 16:04:42 UTC
Created attachment 98242 [details] [review]
Bug 24555: Replace query string with boolean query

Test plan:
 Just make simple and advanced searches
Comment 2 Alex Arnaud 2020-01-31 16:04:49 UTC
Created attachment 98243 [details] [review]
Bug 24555: add a button that shows ES query in search result
Comment 3 Michal Denar 2020-01-31 20:32:35 UTC
Created attachment 98247 [details] [review]
Bug 24555: Replace query string with boolean query

Test plan:
 Just make simple and advanced searches

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 4 Michal Denar 2020-01-31 20:32:39 UTC
Created attachment 98248 [details] [review]
Bug 24555: add a button that shows ES query in search result

Signed-off-by: Michal Denar <black23@gmail.com>
Comment 5 Julian Maurice 2020-05-06 13:24:29 UTC
Created attachment 104419 [details] [review]
Bug 24555: Use 'cross_fields' type for multi_match queries

It allows to search in all fields as if it were one big field.

It also uses 'and' as operator, which means that all searched terms must
appear at least in one field for a document to match

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-multi-match-query.html#type-cross-fields
Comment 6 Julian Maurice 2020-05-06 13:50:16 UTC
Created attachment 104422 [details] [review]
Bug 24555: Use 'cross_fields' type for multi_match queries

It allows to search in all fields as if it were one big field.

It also uses 'and' as operator, which means that all searched terms must
appear at least in one field for a document to match

https://www.elastic.co/guide/en/elasticsearch/reference/6.8/query-dsl-multi-match-query.html#type-cross-fields
Comment 7 Julian Maurice 2020-05-19 08:24:21 UTC
Created attachment 105061 [details] [review]
Bug 24555: Remove 'mc-' prefix in facet field name
Comment 8 Julian Maurice 2020-06-01 11:23:21 UTC
Created attachment 105470 [details] [review]
Bug 24555: Use operator "and" for match query
Comment 9 Julian Maurice 2020-06-01 11:23:55 UTC
Created attachment 105471 [details] [review]
Bug 24555: Fix search limits
Comment 10 Julian Maurice 2020-06-01 12:04:52 UTC
Created attachment 105472 [details] [review]
Bug 24555: Fix search limits
Comment 11 Julian Maurice 2020-06-11 12:37:20 UTC
Created attachment 105744 [details] [review]
Bug 24555: Fix searches like 'field-name:term'

This is used for instance in Koha::Authorities::get_usage_count
("an:<authid>")
Comment 12 Séverine Queune 2021-04-14 10:59:00 UTC
So sad :/

Applying: Bug 24555: Replace query string with boolean query
Using index info to reconstruct a base tree...
M       Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
Falling back to patching base and 3-way merge...
Auto-merging Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
CONFLICT (content): Merge conflict in Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
error: Failed to merge in the changes.
Comment 13 Katrin Fischer 2023-12-31 13:56:32 UTC
What is the status of this?