While resolving some indexing probs for customers using icu in zebra, noted that recommendations for using icu suggested copying phrases-icu.xml from the zerbra install into koha. That procedure is a bit at odds with standard install and upgrade procedure. Need to add a useable phrases-icu.xml to koha and to ensure that install now updates default.idx correctly
Created attachment 20346 [details] [review] Proposed patch Testing: With icu enabled, default.idx will now use icuchain phrases-icu.xml for phrase indexes after a make install or make upgrade. word indexes will state icuchain words-icu.xml The new file will be copied to etc/zebradb/etc by install/upgrade
Created attachment 21045 [details] [review] bug 10729 add phrases configuration for icu
Hi Colin, can you explain what difference this will make in using Koha with ICU indexing? What is the problem and how can I reproduce it?
Hi Katrin, the ICU suppot is mandatory when you mix two or more alphabet in you bibliographic data. The patch of Colin is about phrases indexing. To index ICU needs trasformation, and it is better to have a trasformation for word and a different trasformation for phrases. The path insert a more correct trasformation for phrases.
Hi Zeno, thx for your explanation. Can someone please add a step by step test plan for this?
To test the patch we have done: First time a)Installed it before to do 'perl Makefile.PL' b)Install with ICU c)Catalogue a book d)Search it with 'title phrase' Second time a)Install a master without the patth and Install with ICU b)Install the patch and do a make upgrade, see if the new config is installed This test doesn't show you the difference between previous setup and the patch setup. I think that to see the difference we need an help form Colin
One site found they were getting very inconsistent results between searches. In trying to locate what might be the problem they found that most accounts of how to enable icu in koha on the web suggest copying the files ( words and phrases ) from zebra and editing the configs. i.e. shortcircuiting the koha install/upgrade process (and having to redo your changes after a standard upgrade). Site in question felt that having copied phrases their problems went away, but as the problem was erratic it hard to say conclusively what fixed it. (There is an argument for not stripping whitespace in phrase searches). So I'm not saying that this fixes a specific indexing problem. Its intention was to allow icu to be enabled without having to break the standard install/upgrade procedure. The test is whether with icu enabled the two files words-icu.xml and phrases-icu.xml get installed and default.idx references them on install and preserved on make upgrade
It looks to me like this doesn't touch files relevant for the packages, but wonder if it should - I am adding Robin in CC to take a look.
Should be fine, they get installed by the makefile, and building packages runs that, so they get picked up.
Thx for taking a look Robin.
I am sorry, but this doesn't work for me. My installation uses DOM indexing (authority and bibliographic records) and ICU. What I did: - Reindexed my date, search works correctly - Applied patch - perl Makefile.PL --prev-install-log ../koha-dev/misc/koha-install-log - make upgrade - restarted Zebra server Then tried to reindex: [~/kohaclone (98-10729-icuphrase)⚡]> ./misc/migration_tools/rebuild_zebra.pl -b -v -r Zebra configuration information ================================ Zebra biblio directory = /home/katrin/koha-dev/var/lib/zebradb/biblios Zebra authorities directory = /home/katrin/koha-dev/var/lib/zebradb/authorities Koha directory = /home/katrin/kohaclone BIBLIONUMBER in : 999$c BIBLIOITEMNUMBER in : 999$d ================================ skipping authorities ==================== exporting biblio ==================== 101.................................................................................................. Records exported: 118 ==================== REINDEXING zebra ==================== *** stack smashing detected ***: zebraidx terminated ======= Backtrace: ========= ... lots of unhappiness ... Without the patch the same database reidexes without any problems.
(In reply to Katrin Fischer from comment #11) > I am sorry, but this doesn't work for me. > What versions of yaz and zebra are you using? also what is the output of icu-config --version ?
Hi Colin, I can't check before tonight, but maybe this might give you an idea - I have installed all dependencies using the koha-common packages under Ubuntu.
icu-config wasn't installed. sudo apt-get install libicu-dev icu-config --version 4.8.1.1 Zebra 2.0.44 I was not sure how to get he yaz version, so this is what I did: yaz-client zversion zversion is 3
(In reply to Katrin Fischer from comment #14) > icu-config wasn't installed. > sudo apt-get install libicu-dev > icu-config --version > 4.8.1.1 > Zebra 2.0.44 > I was not sure how to get he yaz version, so this is what I did: > yaz-client > zversion > zversion is 3 I suspect its the bounds checking bug in zebra fixed in 2.0.51. I'll try building an older version to confirm Here's the zebra changelog: --- 2.0.51 2012/05/31 Fix buffer overrun for terms using ICU normalization. Increase max size of indexing term from 256 to 512.
Created attachment 21778 [details] [review] Amended Patch One character change to icu-phrases.xml changes tokenize rule from s to l (wgich matches the values in icu-words.xml. This seems to avoid the buffer overrun in zebra versions before 2.0.50
Thx Colin!
OK, so lets add some background to this bug. We have many customers using both ICU and DOM with zebra, and we found that phrase searching was broken. A phrase search would end up only actually searching upon the first word in the phrase and as such return rather unspecific results. This was reasonably apparent in the clients as 'bad results' in search.. but that's very objective. It's also apparent when using the link bibs to authorities script and this is much more provable. The link bibs to authorities script uses the exact search type that this patch affects, and we found that applying the pathc would drastically improve matching in the default mode of the script. Test plan to follow... once I've got Ian to give me some exact examples.
Thx Martin!
This patch should affect linking of authorities where multiple similar authorities exist, i.e. Law Agency Law Agency Great Britian Law Agency Germany Law Agency France Before the patch, these authorities would all be matched during a search for 'Law Agency' and thus would a link would not be created. With the patch, the full phrase is searched correctly and so an exact match should happen, thus a link would be created. The overall upshot, is that if you have a catalogue liek above, then running link_bibs script sohuld return many more posotive actions after the patch is applied than compared to before the patch is applied
Created attachment 22451 [details] [review] [PASSED QA] Bug 10729: Add phrases configuration for icu Add a separate phrases-icu.xml for phrase indexes The file is based on that distributed with zebra with a couple of additions to reflect Koha usage This patch adds a separate tokenizer variable for phrase indexes so that default.idx is correctly rewritten for sites using icu indexing Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> - Applied patch - perl Makefile.PL --prev-install-log ../koha-dev/misc/koha-install-log - make upgrade - Restarted Zebra server - Did a full reindex of bibliographic and authorities - Checked various searches - Links records to authorities - Checked created links work correctly I couldn't find a regression with this patch. Passes all tests and QA script.
It seems very interesting. Will it be in 3.14 ? Mathieu
Pushed to master. Thanks, Colin!
I still got stack smashing with the updated phrases-icu.xml. My system: Koha 3.12.12 on Ubuntu 12.04 amd64 idzebra-2.0 (2.0.44-3) libyaz4 (4.2.18-1build1) However, if I specify words-icu.xml for phrase index instead of phrases-icu.xml, the indexing will finish fine. Error: Records exported: 476647 ==================== REINDEXING zebra ==================== *** stack smashing detected ***: zebraidx terminated ======= Backtrace: ========= /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x37)[0x7f6b3ce9df47] /lib/x86_64-linux-gnu/libc.so.6(__fortify_fail+0x0)[0x7f6b3ce9df10] /usr/lib/libidzebra-2.0.so.0(+0x17007)[0x7f6b3d4f8007] /usr/lib/libidzebra-2.0.so.0(+0x1712b)[0x7f6b3d4f812b] /usr/lib/libidzebra-2.0.so.0(heap_cread_item2+0xf2)[0x7f6b3d4f82b2] /usr/lib/libidzebra-2.0.so.0(insert_leaf+0x21a)[0x7f6b3d52395a] /usr/lib/libidzebra-2.0.so.0(insert_int+0x756)[0x7f6b3d5249f6] /usr/lib/libidzebra-2.0.so.0(isamb_merge+0xc3)[0x7f6b3d524cf3] /usr/lib/libidzebra-2.0.so.0(heap_inpb+0x193)[0x7f6b3d4f8a23] /usr/lib/libidzebra-2.0.so.0(zebra_index_merge+0x258)[0x7f6b3d4f8e38] /usr/lib/libidzebra-2.0.so.0(+0x298df)[0x7f6b3d50a8df] /usr/lib/libidzebra-2.0.so.0(zebra_end_transaction+0x11d)[0x7f6b3d50e26d] /usr/lib/libidzebra-2.0.so.0(zebra_end_trans+0x31)[0x7f6b3d50e3d1] zebraidx[0x401884] /lib/x86_64-linux-gnu/libc.so.6(__libc_start_main+0xed)[0x7f6b3cdb476d] zebraidx[0x401a0d]
Reading this thread: http://koha.1045719.n5.nabble.com/Stack-Smashing-td3069232.html it seems something connect with Ubuntu. Could you try to test in Debian ?
I did some test by starting with phrases-icu.xml and change it towards words-icu.xml. Here is what I got: 1. Adding the first two transliterate rules in words-icu.xml to phrases-icu.xml. --> Stack Smashed. 2. Changing this line: <transform rule="[:Punctuation:] Remove"/> to <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/> Now it index fine. So, if I just add the WhiteSpace remove rule to phrases-icu.xml, what effect it will cause?
(In reply to Pongtawat from comment #26) > So, if I just add the WhiteSpace remove rule to phrases-icu.xml, what effect it will cause? I think you must open a new bug.
Pushed to 3.14.x, will be in 3.14.08
(In reply to Pongtawat from comment #26) > I did some test by starting with phrases-icu.xml and change it towards > words-icu.xml. Here is what I got: > > 1. Adding the first two transliterate rules in words-icu.xml to > phrases-icu.xml. > --> Stack Smashed. > > 2. Changing this line: > > <transform rule="[:Punctuation:] Remove"/> > > to > > <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/> > > Now it index fine. > > So, if I just add the WhiteSpace remove rule to phrases-icu.xml, what effect > it will cause? Hi, in fact the conf of phrases-icu.xml is still very similar of words-icu.xml. The conf is tested well on a Latin enviroment, with latin special char (è, à, Ł, Ń, etc.). So in a Thai enviroment, like you, probably is better to use a different configuration. An help on setup is here: http://www.indexdata.com/yaz/doc/yaz-icu.html In my opinion is good to have two files, one for words and one for phrase, but many (particulary outsite Europe/America) need to personalize them.
Also have a look at : http://wiki.koha-community.org/wiki/Correcting_Search_of_Arabic_records
(In reply to Fridolin SOMERS from comment #30) > Also have a look at : > http://wiki.koha-community.org/wiki/Correcting_Search_of_Arabic_records I'm not sure that all of the procedure outlined there is necessary. One of the sites we support has records in Arabic, Chinese and Cyrillic and tested the filing of each. Switching to ICU was all that was needed to get these records filing to their satisfaction. The transliterations in the wiki article were not required. Its an area that is unfortunately little documented
Pushed to 3.12.x will be in 3.12.13
(In reply to Galen Charlton from comment #23) > Pushed to master. Thanks, Colin! Patch already included in 3.16.x branch