Bug 19905 - Don't append wildcard to query terms
Summary: Don't append wildcard to query terms
Status: CLOSED WONTFIX
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: David Gustafsson
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-01-02 15:26 UTC by David Gustafsson
Modified: 2018-12-03 20:05 UTC (History)
1 user (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 19905 - Don't append wildcard to query terms (1.63 KB, patch)
2018-01-02 15:52 UTC, David Gustafsson
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Gustafsson 2018-01-02 15:26:25 UTC
Discovered this issue while testing the search links in biblio-details. The links are hard-coded in the template with queries not using quotes for field values, which makes QueryBuilder::_truncate_terms append '*' to the search term. Since wild card searches skips the analyzer-phase, most of our links does not work (since a lot of field values has word boundaries and/or punctuation marks, for example the query "Provider:Taschen," will be converted to "Provider:Taschen,*" and since the indexed value for Provider is "Taschen" this search will produce no results.

In my opinion (if I'm not misunderstanding the purpose of this code) appending '*' to words should be removed.
Comment 1 David Gustafsson 2018-01-02 15:52:42 UTC
Created attachment 70227 [details] [review]
Bug 19905 - Don't append wildcard to query terms

Appending '*' to query terms makes Elasticsearch skip search
analysis. This will for example cause issues with search links
in biblio details for values other than a just single term, since
the indexed value has been tokenized.

How to test:
1) Find a biblio with a field value containing word boundaries and/or
punctuation marks. For example: "au:Author, Name".
2) Perform a search using this query, this should not produce any results.
3) Apply patch.
4) Search using the same query again, it should now produce the expected
results.
Comment 2 David Gustafsson 2018-01-02 15:57:10 UTC
I just now discovered that the link I was testing this with (Publisher, using the query Provider:<value>, is not quoting the value, but most (perhaps all) of the other links are. So an alternative solution to this issue could be add quotes in the xsl-template. Still we have received serveral complaints from libraries failing to perform searches using unquoted field values, and though we could tell everyone to always use quotes, it feels like Koha would provide a better user experience if wildcard queries where not the default.
Comment 3 David Gustafsson 2018-01-02 15:58:19 UTC
*librarians, not libraries.
Comment 4 David Gustafsson 2018-01-02 16:11:58 UTC
Looking through the tests I now saw there is an "QueryAutoTruncate" option that was enabled in my instance. So turning this off will most likely fix this issue for us, and I now realize it cannot just be removed (without also removing this option), making this fix an incorrect one. I have not tested it out myself, but I think there may be other issues with the current autotruncation, should not the whole phrase be searched (the "raw" field for example) or else you will get strange results for wildcard queries containing multiple tokens (since it is the tokanized values that will be matched against)? But that is another possible issue, and should perhaps not be addressed here. But for now I'm disabling the option, and this issues should perhaps be closed.

There is still an issue with the Provider-link not being quoted (and might not work if "QueryAutoTruncate" is enabled), but better to open another issue for that one.
Comment 5 Katrin Fischer 2018-02-19 07:14:16 UTC
Hi David, I see this is marked in discussion - something we can do about it?
Comment 6 David Gustafsson 2018-02-19 09:40:41 UTC
Thanks for the comment. No, not really, this issue can be closed. I do perhaps think it's good to be aware of the fact that analysis phase of ElalistSearch is skipped for wildcard-query, so the QueryAutoTruncate-option is a little bit dangerous in that regard, and should perhaps be recommended against. But better to do something about that in another issue in that case.
Comment 7 Katrin Fischer 2018-02-19 10:21:05 UTC
Hi David, sorry, a bit lost in the discussion, please open a new report for the QueryAutoTruncate issue if you think it makes sense. Closing this one here.