| Summary: | "Begins with" search not possible with ICU indexing | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Katrin Fischer <katrin.fischer> |
| Component: | Searching | Assignee: | Galen Charlton <gmcharlt> |
| Status: | RESOLVED DUPLICATE | QA Contact: | Testopia <testopia> |
| Severity: | normal | ||
| Priority: | P5 - low | CC: | abl, dcook, thanhtutsoetgi |
| Version: | Main | ||
| Hardware: | All | ||
| OS: | All | ||
| See Also: |
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6425 http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11158 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16921 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27299 |
||
| Change sponsored?: | --- | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: | Version(s) released in: | ||
| Circulation function: | |||
|
Description
Katrin Fischer
2015-02-23 11:09:27 UTC
Hm, I have to correct: ti= gives more results than ti,ext= But: results don't match the expected result. I am not really sure what is happening there. On a non-ICU installation something like: ti,ext=Pädagogik Works perfectly, all titles shown start with Pädagogik. The manual: http://manual.koha-community.org/3.18/en/searchguide.html#searchguide-advanced It would be interesting to look at this using yaz-client... I'd be happy to help testing - just not sure how to do it. (In reply to Katrin Fischer from comment #3) > I'd be happy to help testing - just not sure how to do it. 1) Find the unix socket for your zebra database Find "koha-conf.xml" and get the value from <listen id="biblioserver" > e.g. "unix:/blah/blah/zebradb/bibliosocket" 2) Connect to the socket using yaz-client "yaz-client unix:/blah/blah/zebradb/bibliosocket" 3) Select the correct database "base biblios" 3) Run some queries "find @attr 1=4 Pädagogik" "find @attr 1=4 @attr 4=1 @attr 6=3 Pädagogik" @attr 1=4 @attr 4=1 @attr 6=3 is the PQF equivalent of the "ti,ext" CCL, while @attr 1=4 is the same as "ti". 4) Compare the results to see if they return the same results or not -- This should show whether or not these queries are returning the same results. If they are returning the same results, it's a bug in Zebra. If they aren't returning the same results using yaz-client, but they are when using Koha, then it's down to the "extra stuff" that we do to queries before sending them off to Zebra. In that case, you'd need to add some debugging code to see the query before it gets sent off to Zebra. This is probably best done in search.pl or opac-search.pl. For what it's worth, I just tried: Z> base biblios Z> find @attr 1=4 test Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 5, setno 4 SearchResult-1: term=test cnt=5 records returned: 0 Elapsed: 0.000979 Z> find @attr 1=4 @attr 4=1 @attr 6=3 test Sent searchRequest. Received SearchResponse. Search was a success. Number of hits: 0, setno 5 SearchResult-1: term=test cnt=0 records returned: 0 Elapsed: 0.000381 This is using Zebra 2.0.60 from IndexData's Debian repository. -- I just tried in the staff client of Koha 3.8 and got the same results as I did in Zebra. I've tried changing the QueryAutoTruncate, QueryStemming, and QueryWeightFields system preferences to see if they made a difference, but there was no difference. -- Maybe I just don't have the right test string though. What do you mean when you say that "results don't match the expected result"? Can you share the URL of the site where you're experiencing the problem? Hi David,
I am not sure how to interpret your test results. The problem I see is that the search seems to work... but the records titles DON'T BEGIN all with with the search term I used ("Pädagogik..."), when ICU is enabled. I our other installations using chr indexing, this works correctly. It should not depend on the search term.
It would be interesting to see the titles for your results maybe?
(In reply to Katrin Fischer from comment #6) > Hi David, > > I am not sure how to interpret your test results. The problem I see is that > the search seems to work... but the records titles DON'T BEGIN all with with > the search term I used ("Pädagogik..."), when ICU is enabled. I our other > installations using chr indexing, this works correctly. It should not depend > on the search term. > > It would be interesting to see the titles for your results maybe? Ah, I think I wasn't understanding your original question. My test was just to show that a "ti" and a "ti,ext" search weren't returning the same results. In my example, the former returned 5 results while the latter returned 0. I'll have to look more at this another time... (In reply to Katrin Fischer from comment #1) > Hm, I have to correct: > > ti= gives more results than ti,ext= > But: results don't match the expected result. I am not really sure what is > happening there. > > On a non-ICU installation something like: > ti,ext=Pädagogik > Works perfectly, all titles shown start with Pädagogik. I guess Zebra version being used may be somehow relevant. Such searches were indeed not working correctly w/ ICU in some older versions (like 2.0.44, shipped by default with Debian Wheezy). AFAIRC, in particular 'ext' (4=1 6=3) was affected by this, but also other bib-1 structure, position and trucation attributes (or maybe just some specific combinations of thereof ?) were not yet properly supported in 2.0.44 with ICU enabled. Hi Jacek, thx for looking into this. Do you know if it works for newer versions and if it's safe to use a newer version? (In reply to Katrin Fischer from comment #9) > Hi Jacek, thx for looking into this. Do you know if it works for newer > versions and if it's safe to use a newer version? I was using 2.0.60, which seemed to work fine, but I'm not sure I was testing the issue correctly... Suppose it's worth mentioning that the upcoming versions of Debian/Ubuntu will ship with Zebra 2.0.59, which has a few issues as noted by myself and Tomas, I believe. 2.0.59 has a fairly major problem with ICU where strings containing hyphens aren't tokenized properly. That is to say, when the strings are tokenized, only the first token is used and everything after it is discarded. This was fixed in Zebra 2.0.60, but I think OpenSuse is the only Linux distro which has upgraded its packages to 2.0.60... I have tried upgrading to latest Zebra version available (2.0.61). I have also performed forced re-building of the index after upgrade. It doesn't seem to fix this bug at all. Does anyone has some leads on how to fix this problem? It affects not only "Begins with" search but it also affects "Is exactly" search as well. This probably relates to an issue with ICU tokenization that I've covered more here: https://github.com/indexdata/idzebra/issues/24#issuecomment-751260285 Actually, looking at this again, "ti,ext" wouldn't be a "begins with" search. It's a "is exactly" search. So I'm not sure what the problem is here... (In reply to David Cook from comment #14) > Actually, looking at this again, "ti,ext" wouldn't be a "begins with" > search. It's a "is exactly" search. > > So I'm not sure what the problem is here... To my knowledge ext hasn't worked with ICU in forever - or at least as long as we have been using Koha with ICU (2009). This is even documented in the manual (https://koha-community.org/manual/20.11/en/html/searching.html?highlight=ext) (In reply to Katrin Fischer from comment #15) > (In reply to David Cook from comment #14) > > Actually, looking at this again, "ti,ext" wouldn't be a "begins with" > > search. It's a "is exactly" search. > > > > So I'm not sure what the problem is here... > > To my knowledge ext hasn't worked with ICU in forever - or at least as long > as we have been using Koha with ICU (2009). This is even documented in the > manual > (https://koha-community.org/manual/20.11/en/html/searching. > html?highlight=ext) Fortunately, I have a fix for that at Bug 27299 ^_^. It was misconfiguration all along. (In reply to Katrin Fischer from comment #0) > With ICU activated, a "begins with" search like ti,ext= will return the same > results as a ti= search. > > This is a problem for libraries with big collections where a title you look > for can be pretty unspecific and you need a way to do more exact searches. Are you referring to manual/free-hand searches and not searches automatically generated by Koha? If so, then this bug should be resolved by Bug 27299. (In reply to David Cook from comment #17) > (In reply to Katrin Fischer from comment #0) > > With ICU activated, a "begins with" search like ti,ext= will return the same > > results as a ti= search. > > > > This is a problem for libraries with big collections where a title you look > > for can be pretty unspecific and you need a way to do more exact searches. > > Are you referring to manual/free-hand searches and not searches > automatically generated by Koha? > > If so, then this bug should be resolved by Bug 27299. That's good to know and I am actually very happy to see it and some other old glitches resolved! Shoudl we close this MOVED or duplicate? When I commented last night I didn't realize that I was the original bug author - it's been a while :) (In reply to Katrin Fischer from comment #18) > > That's good to know and I am actually very happy to see it and some other > old glitches resolved! > > Shoudl we close this MOVED or duplicate? When I commented last night I > didn't realize that I was the original bug author - it's been a while :) I was also thinking about marking it as duplicate, so I'll go ahead and do that. Yeah, it's been around a long time! We tend to use CHR but we have a library with ICU with some very observant librarians who helped me to unravel this one. *** This bug has been marked as a duplicate of bug 27299 *** |