Bug 13751 - "Begins with" search not possible with ICU indexing
Summary: "Begins with" search not possible with ICU indexing
Status: RESOLVED DUPLICATE of bug 27299
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-02-23 11:09 UTC by Katrin Fischer
Modified: 2021-01-05 22:28 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2015-02-23 11:09:27 UTC
With ICU activated, a "begins with" search like ti,ext= will return the same results as a ti= search.

This is a problem for libraries with big collections where a title you look for can be pretty unspecific and you need a way to do more exact searches.
Comment 1 Katrin Fischer 2015-02-23 11:35:02 UTC
Hm, I have to correct:

ti= gives more results than ti,ext=
But: results don't match the expected result. I am not really sure what is happening there.

On a non-ICU installation something like:
ti,ext=Pädagogik
Works perfectly, all titles shown start with Pädagogik.

The manual:
http://manual.koha-community.org/3.18/en/searchguide.html#searchguide-advanced
Comment 2 David Cook 2015-02-24 00:17:25 UTC
It would be interesting to look at this using yaz-client...
Comment 3 Katrin Fischer 2015-02-24 08:11:10 UTC
I'd be happy to help testing - just not sure how to do it.
Comment 4 David Cook 2015-02-25 03:54:05 UTC
(In reply to Katrin Fischer from comment #3)
> I'd be happy to help testing - just not sure how to do it.

1) Find the unix socket for your zebra database

Find "koha-conf.xml" and get the value from <listen id="biblioserver" >

e.g. "unix:/blah/blah/zebradb/bibliosocket"

2) Connect to the socket using yaz-client

"yaz-client unix:/blah/blah/zebradb/bibliosocket"

3) Select the correct database

"base biblios"

3) Run some queries

"find @attr 1=4 Pädagogik"
"find @attr 1=4 @attr 4=1 @attr 6=3 Pädagogik"

@attr 1=4 @attr 4=1 @attr 6=3 is the PQF equivalent of the "ti,ext" CCL, while @attr 1=4 is the same as "ti".

4) Compare the results to see if they return the same results or not

--

This should show whether or not these queries are returning the same results.

If they are returning the same results, it's a bug in Zebra.

If they aren't returning the same results using yaz-client, but they are when using Koha, then it's down to the "extra stuff" that we do to queries before sending them off to Zebra.

In that case, you'd need to add some debugging code to see the query before it gets sent off to Zebra. This is probably best done in search.pl or opac-search.pl.
Comment 5 David Cook 2015-02-25 04:02:41 UTC
For what it's worth, I just tried:

Z> base biblios
Z> find @attr 1=4 test
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 5, setno 4
SearchResult-1: term=test cnt=5
records returned: 0
Elapsed: 0.000979
Z> find @attr 1=4 @attr 4=1 @attr 6=3 test
Sent searchRequest.
Received SearchResponse.
Search was a success.
Number of hits: 0, setno 5
SearchResult-1: term=test cnt=0
records returned: 0
Elapsed: 0.000381

This is using Zebra 2.0.60 from IndexData's Debian repository.

--

I just tried in the staff client of Koha 3.8 and got the same results as I did in Zebra.

I've tried changing the QueryAutoTruncate, QueryStemming, and QueryWeightFields system preferences to see if they made a difference, but there was no difference.

--

Maybe I just don't have the right test string though.

What do you mean when you say that "results don't match the expected result"?

Can you share the URL of the site where you're experiencing the problem?
Comment 6 Katrin Fischer 2015-03-13 15:43:02 UTC
Hi David,

I am not sure how to interpret your test results. The problem I see is that the search seems to work... but the records titles DON'T BEGIN all with with the search term I used ("Pädagogik..."), when ICU is enabled. I our other installations using chr indexing, this works correctly. It should not depend on the search term.

It would be interesting to see the titles for your results maybe?
Comment 7 David Cook 2015-03-16 00:24:06 UTC
(In reply to Katrin Fischer from comment #6)
> Hi David,
> 
> I am not sure how to interpret your test results. The problem I see is that
> the search seems to work... but the records titles DON'T BEGIN all with with
> the search term I used ("Pädagogik..."), when ICU is enabled. I our other
> installations using chr indexing, this works correctly. It should not depend
> on the search term.
> 
> It would be interesting to see the titles for your results maybe?

Ah, I think I wasn't understanding your original question.

My test was just to show that a "ti" and a "ti,ext" search weren't returning the same results. In my example, the former returned 5 results while the latter returned 0.

I'll have to look more at this another time...
Comment 8 Jacek Ablewicz 2015-03-16 07:16:29 UTC
(In reply to Katrin Fischer from comment #1)
> Hm, I have to correct:
> 
> ti= gives more results than ti,ext=
> But: results don't match the expected result. I am not really sure what is
> happening there.
> 
> On a non-ICU installation something like:
> ti,ext=Pädagogik
> Works perfectly, all titles shown start with Pädagogik.

I guess Zebra version being used may be somehow relevant. Such searches were indeed not working correctly w/ ICU in some older versions (like 2.0.44, shipped by default with Debian Wheezy). AFAIRC, in particular 'ext' (4=1 6=3) was affected by this, but also other bib-1 structure, position and trucation attributes (or maybe just some specific combinations of thereof ?) were not yet properly supported in 2.0.44 with ICU enabled.
Comment 9 Katrin Fischer 2015-03-16 07:24:10 UTC
Hi Jacek, thx for looking into this. Do you know if it works for newer versions and if it's safe to use a newer version?
Comment 10 David Cook 2015-03-17 01:34:23 UTC
(In reply to Katrin Fischer from comment #9)
> Hi Jacek, thx for looking into this. Do you know if it works for newer
> versions and if it's safe to use a newer version?

I was using 2.0.60, which seemed to work fine, but I'm not sure I was testing the issue correctly...
Comment 11 David Cook 2015-03-17 01:36:54 UTC
Suppose it's worth mentioning that the upcoming versions of Debian/Ubuntu will ship with Zebra 2.0.59, which has a few issues as noted by myself and Tomas, I believe.

2.0.59 has a fairly major problem with ICU where strings containing hyphens aren't tokenized properly. That is to say, when the strings are tokenized, only the first token is used and everything after it is discarded. This was fixed in Zebra 2.0.60, but I think OpenSuse is the only Linux distro which has upgraded its packages to 2.0.60...
Comment 12 Than H. S. (Punsarn.asia) 2015-11-25 08:31:09 UTC
I have tried upgrading to latest Zebra version available (2.0.61). I have also performed forced re-building of the index after upgrade. It doesn't seem to fix this bug at all.

Does anyone has some leads on how to fix this problem? It affects not only "Begins with" search but it also affects "Is exactly" search as well.
Comment 13 David Cook 2021-01-03 23:58:15 UTC
This probably relates to an issue with ICU tokenization that I've covered more here: https://github.com/indexdata/idzebra/issues/24#issuecomment-751260285
Comment 14 David Cook 2021-01-04 00:27:38 UTC
Actually, looking at this again, "ti,ext" wouldn't be a "begins with" search. It's a "is exactly" search. 

So I'm not sure what the problem is here...
Comment 15 Katrin Fischer 2021-01-04 08:11:36 UTC
(In reply to David Cook from comment #14)
> Actually, looking at this again, "ti,ext" wouldn't be a "begins with"
> search. It's a "is exactly" search. 
> 
> So I'm not sure what the problem is here...

To my knowledge ext hasn't worked with ICU in forever - or at least as long as we have been using Koha with ICU (2009). This is even documented in the manual (https://koha-community.org/manual/20.11/en/html/searching.html?highlight=ext)
Comment 16 David Cook 2021-01-05 00:02:13 UTC
(In reply to Katrin Fischer from comment #15)
> (In reply to David Cook from comment #14)
> > Actually, looking at this again, "ti,ext" wouldn't be a "begins with"
> > search. It's a "is exactly" search. 
> > 
> > So I'm not sure what the problem is here...
> 
> To my knowledge ext hasn't worked with ICU in forever - or at least as long
> as we have been using Koha with ICU (2009). This is even documented in the
> manual
> (https://koha-community.org/manual/20.11/en/html/searching.
> html?highlight=ext)

Fortunately, I have a fix for that at Bug 27299 ^_^. It was misconfiguration all along.
Comment 17 David Cook 2021-01-05 00:02:56 UTC
(In reply to Katrin Fischer from comment #0)
> With ICU activated, a "begins with" search like ti,ext= will return the same
> results as a ti= search.
> 
> This is a problem for libraries with big collections where a title you look
> for can be pretty unspecific and you need a way to do more exact searches.

Are you referring to manual/free-hand searches and not searches automatically generated by Koha?

If so, then this bug should be resolved by Bug 27299.
Comment 18 Katrin Fischer 2021-01-05 08:36:56 UTC
(In reply to David Cook from comment #17)
> (In reply to Katrin Fischer from comment #0)
> > With ICU activated, a "begins with" search like ti,ext= will return the same
> > results as a ti= search.
> > 
> > This is a problem for libraries with big collections where a title you look
> > for can be pretty unspecific and you need a way to do more exact searches.
> 
> Are you referring to manual/free-hand searches and not searches
> automatically generated by Koha?
> 
> If so, then this bug should be resolved by Bug 27299.

That's good to know and I am actually very happy to see it and some other old glitches resolved!

Shoudl we close this MOVED or duplicate? When I commented last night I didn't realize that I was the original bug author - it's been a while :)
Comment 19 David Cook 2021-01-05 22:27:49 UTC
(In reply to Katrin Fischer from comment #18)
> 
> That's good to know and I am actually very happy to see it and some other
> old glitches resolved!
> 
> Shoudl we close this MOVED or duplicate? When I commented last night I
> didn't realize that I was the original bug author - it's been a while :)

I was also thinking about marking it as duplicate, so I'll go ahead and do that.

Yeah, it's been around a long time! We tend to use CHR but we have a library with ICU with some very observant librarians who helped me to unravel this one.
Comment 20 David Cook 2021-01-05 22:28:02 UTC

*** This bug has been marked as a duplicate of bug 27299 ***