Bug 16581 - ICU tokenization bug in idzebra-2.0 2.0.59-1
Summary: ICU tokenization bug in idzebra-2.0 2.0.59-1
Status: REOPENED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL: http://ftp.indexdata.dk/pub/zebra/deb...
Keywords: dependency
Depends on:
Blocks:
 
Reported: 2016-05-24 21:57 UTC by Barton Chittenden
Modified: 2018-03-12 11:53 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Barton Chittenden 2016-05-24 21:57:24 UTC
The version of idzebra-2.0

idzebra-2.0 2.0.59-1

which is the version in the debian-jesse repository, has a known ICU-Tokenization bug:

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=777515

This has been seen to cause search errors with, for instance, the Chinese language search for 心意更新. Searches for the string of all four characters, without spaces, will give hundreds of hits, whereas separating the ideograms with spaces will limit the search to 8 search results, as intended.
Comment 1 David Cook 2016-05-24 23:32:05 UTC
Hi Barton,

I was the one who found that bug in Zebra last year, and while I got Indexdata to fix it in 2.0.60 (http://www.indexdata.com/zebra/doc/NEWS https://github.com/indexdata/idzebra/commit/704fd190292cb771df94553b0ed6f9f4b71660a6), I'm not sure that's the same issue that you're having. 

Have you tested your query with newer versions of Zebra (2.0.60+) to see if the issue still exists in newer versions?
Comment 2 David Cook 2016-05-24 23:34:45 UTC
Robin and I mentioned this bug between Feb-May 2015 but no one really seemed to care at the time (http://permalink.gmane.org/gmane.education.libraries.koha.devel/11966).

I don't know if that means that people are using the Indexdata repositories or if they're just ignoring erroneous behaviour in Zebra when using ICU...
Comment 3 M. Tompsett 2016-08-22 20:56:46 UTC
(In reply to David Cook from comment #2)
> I don't know if that means that people are using the Indexdata repositories
> or if they're just ignoring erroneous behaviour in Zebra when using ICU...

Probably the latter. :)
Comment 4 Zeno Tajoli 2016-08-25 09:15:53 UTC
In fact I use ICU on jessie and I install Zebra from  Indexdata repositories.
I think it is the only option avaible.
Comment 5 M. Tompsett 2016-08-25 11:32:44 UTC
I've added a link to the IndexData Debian README file into the URL field above.
The Ubuntu one is here: http://ftp.indexdata.dk/pub/zebra/ubuntu/README
Comment 6 David Cook 2016-08-30 23:03:51 UTC
(In reply to Zeno Tajoli from comment #4)
> In fact I use ICU on jessie and I install Zebra from  Indexdata repositories.
> I think it is the only option avaible.

Agreed.

I don't know the Koha community's official position on the subject, but I think we should be advocating the use of IndexData's repositories or providing a fully functional Zebra package ourselves. Relying on the Debian repository is clearly not a good option :/.
Comment 7 Barton Chittenden 2016-08-31 13:39:38 UTC
09:34 < barton> I'm looking at bug 16581... I think there's still a number of un-answered questions there...
09:34 < huginn> Bug http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16581 enhancement, P5 - low, ---, 
                gmcharlt, RESOLVED WONTFIX, ICU tokenization bug in idzebra-2.0 2.0.59-1
09:35 < marcelr> barton: i did not understand why it was closed either

09:36 < barton> so here are the things that I'm wondering about: 1) is it possible to require the use of the 
                indexdata repo? 2) what version of idzebra-2.0 *should* we be using?
09:37 < mtompset> 1) why not? It's a good idea, 2) the current one -- I think something 62?
Comment 8 Katrin Fischer 2016-08-31 20:57:27 UTC
Adding Mirko to the discussion.
Comment 9 David Cook 2016-09-04 23:53:04 UTC
Overall, I'm in favour of using Indexdata's repository. 

As Mark pointed out, 2.0.62 appears to be the latest version: http://www.indexdata.com/zebra/doc/NEWS

You'll notice that 2.0.60 is where the tokenization bug was fixed: Fix ICU phrase searches for terms split by ICU ZEB-664

You'll probably notice a big gap between 2.0.59 and 2.0.60. My emails to Indexdata prompted the fix for 2.0.60. You may also notice "Allow @attr 2=102 for numeric index" in 2.0.62. I had noticed that relevance ranking didn't work with a query containing a numeric index, so Indexdata fixed it. I don't think it really affects the ranking much per se, but it prevents the Zebra error. 

Note also that you an now browse Zebra's code on Github: https://github.com/indexdata/idzebra

I don't know if they accept pull requests, but I wouldn't be surprised if they did.

Of course, I don't know if Indexdata will be doing much work on Zebra going forward. I don't think they're answering the Zebra-list anymore: http://lists.indexdata.dk/pipermail/zebralist/2016-May/002645.html

I think they're focusing their efforts on the FOLIO project these days. Tomas mentioned some time ago that Indexdata considered Zebra a legacy project, and that seems to be the case.

All the more reason to move to Elastic, I suppose, although I see the utility of keeping Zebra around if only as a Z39.50 server. Makes you wonder if we should be thinking about turning Zebra into an optional component once Elastic takes over fully...
Comment 10 Fridolin SOMERS 2018-03-12 11:53:51 UTC
(In reply to David Cook from comment #9)
> Overall, I'm in favour of using Indexdata's repository. 
>
We at Biblibre are using this repo with Ubuntu Xenial since Koha 16.11.
Its Zebra 2.1.3 now.