Bug 38749 - Truncation doesn't work as expected with phrase searches
Summary: Truncation doesn't work as expected with phrase searches
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-12-19 01:46 UTC by David Cook
Modified: 2024-12-24 00:15 UTC (History)
1 user (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2024-12-19 01:46:20 UTC
If you do a "Keyword as phrase" search in Koha when using Zebra, a search for 'testa testb*' will turn into this:

(rk=(kw,phr: testa and kw,phr,rtrn:testb ))

This may or may not make sense when doing a wrdl or word search, but it certainly doesn't make sense when doing a phr phrase search, because it breaks up the terms in the phrase you were searching.

That said... Zebra doesn't support * for doing a masked/wildcard search, so it's not clear at a glance how a person could fix this.
Comment 1 David Cook 2024-12-19 01:50:06 UTC
However, there is a way. 

If you comment out all our current truncation detection code and do the following...

                 if ($auto_truncation){
-                        unless ( $index =~ /,(st-|phr|ext)/ ) {
+                        unless ( $index =~ /,(st-|ext)/ ) {

+                my $trunc_count = ($operand =~ s/\Q*\E/#/g);
+                if ($trunc_count){
+                    $index .= ",process-in-search-term";
+                }

Zebra does support # as a mask/wildcard in lieu of *. 

It does mean it would be wise to s/#//g the operand before doing truncation handling, so that "C# Program*" doesn't match "CGI Programming". 

But that wouldn't be that hard to do.

The # would work with both left and right truncation.

--

Anyway, food for thought. At this point, I don't think that we're going to pursue this change. Something like 99% of our libraries are using Elasticsearch now. 

But it's an idea...
Comment 2 Fridolin Somers 2024-12-19 09:04:28 UTC
> Zebra doesn't support * for doing a masked/wildcard search

Strange I never noticed this.
Looks correct indeed in CCL doc :
https://software.indexdata.com/yaz/doc/tools.html#ccl.syntax

Not sure where to start on this complex topic
Comment 3 David Cook 2024-12-19 22:23:42 UTC
(In reply to Fridolin Somers from comment #2)
> > Zebra doesn't support * for doing a masked/wildcard search
> 
> Strange I never noticed this.
> Looks correct indeed in CCL doc :
> https://software.indexdata.com/yaz/doc/tools.html#ccl.syntax

I hadn't thought about the CCL syntax specifically. That's interesting.

"2.4.4. Truncation Attributes (type = 5)" on https://software.indexdata.com/zebra/doc/querymodel-rpn.html

Our Koha CCL qualifier "process-in-search-term" maps to "Process # in search term" for the RPN truncation.

I couldn't get that CCL syntax to work but the RPN one does. In CCL, the # is a mask for one character whereas in the RPN it's translated into .* and then does a regexp search. 

Although as I say that... I wonder if that means other characters in the query that could be a regular expression character would also be interpreted. Probably. 

> Not sure where to start on this complex topic

Yeah, that's why I'm thinking of not pursuing this one. Zebra is on the way out, and changing a search fundamental seems... problematic, even if it could be an improvement.
Comment 4 David Cook 2024-12-19 23:03:52 UTC
Elasticsearch... "kw,phr" doesn't actually do anything it seems. It just gets stripped out of the query.

When I use Elasticsearch with Koha, I can't seem to get wildcards like * to work instead of double quoted phrases.

According to the Internet... it's not clear whether or not wildcards should work in phrase searches. Some things suggest yes and some things suggest no.

Koha query: kw,phr:"handb*"
Query sent to Elastic: (""handb*"")
0 results

Koha query: "hand*"
Query sent to Elastic: ("hand*")
2 results

Koha query: "handb*"
Query sent to Elastic: (""handb*"")
0 results

No idea why handb* gets bad double quoting whereas hand* is fine. Guessing we have some code that goes off the term length or something...

Anyway, the following curl suggests that wildcards don't work in ES phrase searches when using the ES DSL:

curl -u <username:password> -X POST "<ES server>" -H 'Content-Type: application/json' -d '{"query": {"query_string": {"query": "handb*","analyze_wildcard": true}}}'

(removed the ES server link so I don't get autobanned by BZ like usual)
Comment 5 David Cook 2024-12-19 23:05:53 UTC
(In reply to David Cook from comment #4)
> curl -u <username:password> -X POST "<ES server>" -H 'Content-Type:
> application/json' -d '{"query": {"query_string": {"query":
> "handb*","analyze_wildcard": true}}}'

Sorry that's the non-phrase which does work and gets 4 results.

This is the phrase query which gets 0 results:

-d '{"query": {"query_string": {"query": "\"handb*\"","analyze_wildcard": true}}}'

This phrase query will get 3 results though:
-d '{"query": {"query_string": {"query": "\"handbook\"","analyze_wildcard": true}}}'
Comment 6 David Cook 2024-12-19 23:31:52 UTC
I'm thinking that maybe we just add a warning to the syspref "QueryAutoTruncate" saying that truncation doesn't work for phrase searches for either Zebra or Elasticsearch.

(Of course, we should probably also remove the "X as phrase" advanced search options when using Elasticsearch since they don't actually do anything...)
Comment 7 David Cook 2024-12-20 00:07:19 UTC
I was hopeful that I could use the regular expression syntax for the ES query string syntax, but alas... we're mangling the ES queries...

Koha: title:/handb.*/
Sent to elastic: ("title:/handb.*/")
0 results

I sent this via curl:
"query": "title:/handb.*/"
2 results

--

Admittedly, the regular expressions in ES don't seem to be able to handle multiple terms. If you try "title:/a handb.*/" or "title:/handb.* of/" you'll get 0 results unfortunately.
Comment 8 David Cook 2024-12-20 00:15:34 UTC
Of course, allowing regular expressions also comes with peril. So that's fun too.
Comment 9 Katrin Fischer 2024-12-20 08:11:26 UTC
(In reply to David Cook from comment #6)
> I'm thinking that maybe we just add a warning to the syspref
> "QueryAutoTruncate" saying that truncation doesn't work for phrase searches
> for either Zebra or Elasticsearch.
> 
> (Of course, we should probably also remove the "X as phrase" advanced search
> options when using Elasticsearch since they don't actually do anything...)

Are you saying phrase search doesn't work at all or only not working as expected with QueryAutoTruncate?
Comment 10 David Cook 2024-12-22 23:10:54 UTC
(In reply to Katrin Fischer from comment #9)
> (In reply to David Cook from comment #6)
> > I'm thinking that maybe we just add a warning to the syspref
> > "QueryAutoTruncate" saying that truncation doesn't work for phrase searches
> > for either Zebra or Elasticsearch.
> > 
> > (Of course, we should probably also remove the "X as phrase" advanced search
> > options when using Elasticsearch since they don't actually do anything...)
> 
> Are you saying phrase search doesn't work at all or only not working as
> expected with QueryAutoTruncate?

It's complicated haha.

Zebra: If you use "phr" with * but don't use double-quotes, Koha mangles the query and separates the phrase into individual terms that are searched against the "phrase" register. Technically truncation does apply to the terms, but it's essentially broken the phrase search, so it's not really a phrase search anymore.

If you use "phr" with * and use double quotes, Koha keeps it as a phrase search, but the truncation won't work. 

Elasticsearch: "phr" just gets erased from the query, so it's a useless adding it to a query in Koha.

Elasticsearch can only use * with individual terms. If you use double quotes, which is ES's Query String Syntax for a phrase search, it will silently ignore the * and won't perform the truncation.
Comment 11 Katrin Fischer 2024-12-23 08:35:05 UTC
It sure sounds complicated.

Would it make sense then to ignore * on phrase searches as that would at least phrase search?
Comment 12 David Cook 2024-12-24 00:15:26 UTC
(In reply to Katrin Fischer from comment #11)
> It sure sounds complicated.
> 
> Would it make sense then to ignore * on phrase searches as that would at
> least phrase search?

For Zebra, that could make sense.

For Elastic, I wonder if we should be trying to enforce double quotes when the phr index is used... more thinking to do one that one.