If you do a "Keyword as phrase" search in Koha when using Zebra, a search for 'testa testb*' will turn into this: (rk=(kw,phr: testa and kw,phr,rtrn:testb )) This may or may not make sense when doing a wrdl or word search, but it certainly doesn't make sense when doing a phr phrase search, because it breaks up the terms in the phrase you were searching. That said... Zebra doesn't support * for doing a masked/wildcard search, so it's not clear at a glance how a person could fix this.
However, there is a way. If you comment out all our current truncation detection code and do the following... if ($auto_truncation){ - unless ( $index =~ /,(st-|phr|ext)/ ) { + unless ( $index =~ /,(st-|ext)/ ) { + my $trunc_count = ($operand =~ s/\Q*\E/#/g); + if ($trunc_count){ + $index .= ",process-in-search-term"; + } Zebra does support # as a mask/wildcard in lieu of *. It does mean it would be wise to s/#//g the operand before doing truncation handling, so that "C# Program*" doesn't match "CGI Programming". But that wouldn't be that hard to do. The # would work with both left and right truncation. -- Anyway, food for thought. At this point, I don't think that we're going to pursue this change. Something like 99% of our libraries are using Elasticsearch now. But it's an idea...
> Zebra doesn't support * for doing a masked/wildcard search Strange I never noticed this. Looks correct indeed in CCL doc : https://software.indexdata.com/yaz/doc/tools.html#ccl.syntax Not sure where to start on this complex topic
(In reply to Fridolin Somers from comment #2) > > Zebra doesn't support * for doing a masked/wildcard search > > Strange I never noticed this. > Looks correct indeed in CCL doc : > https://software.indexdata.com/yaz/doc/tools.html#ccl.syntax I hadn't thought about the CCL syntax specifically. That's interesting. "2.4.4. Truncation Attributes (type = 5)" on https://software.indexdata.com/zebra/doc/querymodel-rpn.html Our Koha CCL qualifier "process-in-search-term" maps to "Process # in search term" for the RPN truncation. I couldn't get that CCL syntax to work but the RPN one does. In CCL, the # is a mask for one character whereas in the RPN it's translated into .* and then does a regexp search. Although as I say that... I wonder if that means other characters in the query that could be a regular expression character would also be interpreted. Probably. > Not sure where to start on this complex topic Yeah, that's why I'm thinking of not pursuing this one. Zebra is on the way out, and changing a search fundamental seems... problematic, even if it could be an improvement.
Elasticsearch... "kw,phr" doesn't actually do anything it seems. It just gets stripped out of the query. When I use Elasticsearch with Koha, I can't seem to get wildcards like * to work instead of double quoted phrases. According to the Internet... it's not clear whether or not wildcards should work in phrase searches. Some things suggest yes and some things suggest no. Koha query: kw,phr:"handb*" Query sent to Elastic: (""handb*"") 0 results Koha query: "hand*" Query sent to Elastic: ("hand*") 2 results Koha query: "handb*" Query sent to Elastic: (""handb*"") 0 results No idea why handb* gets bad double quoting whereas hand* is fine. Guessing we have some code that goes off the term length or something... Anyway, the following curl suggests that wildcards don't work in ES phrase searches when using the ES DSL: curl -u <username:password> -X POST "<ES server>" -H 'Content-Type: application/json' -d '{"query": {"query_string": {"query": "handb*","analyze_wildcard": true}}}' (removed the ES server link so I don't get autobanned by BZ like usual)
(In reply to David Cook from comment #4) > curl -u <username:password> -X POST "<ES server>" -H 'Content-Type: > application/json' -d '{"query": {"query_string": {"query": > "handb*","analyze_wildcard": true}}}' Sorry that's the non-phrase which does work and gets 4 results. This is the phrase query which gets 0 results: -d '{"query": {"query_string": {"query": "\"handb*\"","analyze_wildcard": true}}}' This phrase query will get 3 results though: -d '{"query": {"query_string": {"query": "\"handbook\"","analyze_wildcard": true}}}'
I'm thinking that maybe we just add a warning to the syspref "QueryAutoTruncate" saying that truncation doesn't work for phrase searches for either Zebra or Elasticsearch. (Of course, we should probably also remove the "X as phrase" advanced search options when using Elasticsearch since they don't actually do anything...)
I was hopeful that I could use the regular expression syntax for the ES query string syntax, but alas... we're mangling the ES queries... Koha: title:/handb.*/ Sent to elastic: ("title:/handb.*/") 0 results I sent this via curl: "query": "title:/handb.*/" 2 results -- Admittedly, the regular expressions in ES don't seem to be able to handle multiple terms. If you try "title:/a handb.*/" or "title:/handb.* of/" you'll get 0 results unfortunately.
Of course, allowing regular expressions also comes with peril. So that's fun too.
(In reply to David Cook from comment #6) > I'm thinking that maybe we just add a warning to the syspref > "QueryAutoTruncate" saying that truncation doesn't work for phrase searches > for either Zebra or Elasticsearch. > > (Of course, we should probably also remove the "X as phrase" advanced search > options when using Elasticsearch since they don't actually do anything...) Are you saying phrase search doesn't work at all or only not working as expected with QueryAutoTruncate?
(In reply to Katrin Fischer from comment #9) > (In reply to David Cook from comment #6) > > I'm thinking that maybe we just add a warning to the syspref > > "QueryAutoTruncate" saying that truncation doesn't work for phrase searches > > for either Zebra or Elasticsearch. > > > > (Of course, we should probably also remove the "X as phrase" advanced search > > options when using Elasticsearch since they don't actually do anything...) > > Are you saying phrase search doesn't work at all or only not working as > expected with QueryAutoTruncate? It's complicated haha. Zebra: If you use "phr" with * but don't use double-quotes, Koha mangles the query and separates the phrase into individual terms that are searched against the "phrase" register. Technically truncation does apply to the terms, but it's essentially broken the phrase search, so it's not really a phrase search anymore. If you use "phr" with * and use double quotes, Koha keeps it as a phrase search, but the truncation won't work. Elasticsearch: "phr" just gets erased from the query, so it's a useless adding it to a query in Koha. Elasticsearch can only use * with individual terms. If you use double quotes, which is ES's Query String Syntax for a phrase search, it will silently ignore the * and won't perform the truncation.
It sure sounds complicated. Would it make sense then to ignore * on phrase searches as that would at least phrase search?
(In reply to Katrin Fischer from comment #11) > It sure sounds complicated. > > Would it make sense then to ignore * on phrase searches as that would at > least phrase search? For Zebra, that could make sense. For Elastic, I wonder if we should be trying to enforce double quotes when the phr index is used... more thinking to do one that one.