---- Reported by katrin.fischer@bsz-bw.de 2010-04-22 12:31:31 ---- Created an attachment scan index with fußball When using scan index in advanced search with Fußball the ß is not displayed correctly. Fußball (soccer) -> Fu@@ball Umlauts are reduced to the basic character (ä to a, ü to u). Währung (currency) -> Wahrung (see screenshots) ---- Additional Comments From katrin.fischer@bsz-bw.de 2010-04-22 12:32:39 ---- Created an attachment scan index with Währung (umlaut) ---- Additional Comments From magnus@enger.priv.no 2010-04-22 13:17:22 ---- Created an attachment Norwegian trouble with scan ---- Additional Comments From magnus@enger.priv.no 2010-04-22 13:18:29 ---- I see similar problems with Norwegian æøå, which give these conversions: æ -> @@ ø -> o å -> a --- Bug imported by chris@bigballofwax.co.nz 2010-05-21 01:26 UTC --- This bug was previously known as _bug_ 4397 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4397 Imported an attachment (id=2068) Imported an attachment (id=2069) Imported an attachment (id=2070) Actual time not defined. Setting to 0.0 Setting qa contact to the default for this product. This bug either had no qa contact or an invalid one. The original submitter of attachment 2068 [details] is unknown. Reassigning to the person who moved it here: chris@bigballofwax.co.nz. The original submitter of attachment 2069 [details] is unknown. Reassigning to the person who moved it here: chris@bigballofwax.co.nz. The original submitter of attachment 2070 [details] is unknown. Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
Scan shows Zebra internal datas. I think it is normal to see characters the way the are indexed, no the way they are in record. Transformations depends on word-phrase-utf.chr config file.
I reproduced the problem and using yaz-client I noticed it was a Koha problem actually: Z> open unix:/home/tomas/koha-dev/var/run/zebradb/bibliosocket Connecting...OK. Sent initrequest. Connection accepted by v3 target. ID : 81 Name : Zebra Information Server/GFS/YAZ Version: 4.2.30 98864b44c654645bc16b2c54f822dc2e45a93031 Options: search present delSet triggerResourceCtrl scan sort extendedServices namedResultSets Elapsed: 0.002084 Z> base biblios Z> scan @attr 1=4 @attr 4=6 @attr 5=1 t query: @attr 1=4 @attr 4=6 @attr 5=1 t Received ScanResponse 20 entries, position=1 * t (60) tử (1) tạo (1) tập (3) ta (8) tablature (2) table (3) Tack (1) tacka (1) Tadeusz (2) Tage (2) Tagen (3) Taha (1) tài (2) taide (2) Tain (1) Tainted (1) taiteen (1) taiteokset (1) Takashi (18) Elapsed: 0.316274 Z> while using Koha returned the @@@ instead, as Katrina explained in the bug description. So => ASSIGNED.
Created attachment 25742 [details] [review] Bug 4397: Scan index search results broken for non-latin characters The Zoom specification defines that a ScanSet should provide a way to retrieve terms suitable for displaying and another one for using on further searches [1]. The Net::Z3950::ZOOM implementation actually provides both [2] but we where using the wrong one. Using $scanset->display_term(...) instead of $scanset->term(...) fixes the problem. To test: - Do a index scan search (advanced search > more options > check 'index scan') - Notice non-latin characters are replaced by one or more '@' symbols. - Apply the patch - Re-do the search, everything shows as it should. - Try to follow any of the terms (clicking on them) and notice that it actually gives you relevant results (i.e. is not searching for @!!!!). - Sign off :-D Regards To+ [1] http://zoom.z3950.org/api/zoom-1.4.html#3.6.3 [2] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#term()_/_display_term() Sponsored-by: Universidad Nacional de Cordoba
Created attachment 25817 [details] [review] [Signed-off] Bug 4397: Scan index search results broken for non-latin characters The Zoom specification defines that a ScanSet should provide a way to retrieve terms suitable for displaying and another one for using on further searches [1]. The Net::Z3950::ZOOM implementation actually provides both [2] but we where using the wrong one. Using $scanset->display_term(...) instead of $scanset->term(...) fixes the problem. To test: - Do a index scan search (advanced search > more options > check 'index scan') - Notice non-latin characters are replaced by one or more '@' symbols. - Apply the patch - Re-do the search, everything shows as it should. - Try to follow any of the terms (clicking on them) and notice that it actually gives you relevant results (i.e. is not searching for @!!!!). - Sign off :-D Regards To+ [1] http://zoom.z3950.org/api/zoom-1.4.html#3.6.3 [2] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#term()_/_display_term() Sponsored-by: Universidad Nacional de Cordoba Followed test plan. Patch behaves as expected. Signed-off-by: Marc Véron <veron@veron.ch>
I have problems applying this patch with git bz... $ git bz apply 4397 Traceback (most recent call last): File "/usr/sbin/git-bz", line 2609, in <module> do_apply(*args) File "/usr/sbin/git-bz", line 1688, in do_apply print "Bug %d - %s" % (bug.id, bug.short_desc) UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 38: ordinal not in range(128)
But works fine using wget and git am -iu3...
Right now, I have problems reproducing the initial issue. Arabic and ß display fine for me on master without the patch, umlauts are still losing their diacritic, ü showing as u. Could it be that this works only for some specific setting? Also if you pick "Title" from the scan search options, it explodes into a ZOOM error before and after applying the patch. Waiting for Feedback from Tomas.
This patch doesn't break anything for me, but I have problems to replicate the original display problem on my installations now. Could someone else give this a try?
Created attachment 26470 [details] [review] Bug 4397: Scan index search results broken for non-latin characters The Zoom specification defines that a ScanSet should provide a way to retrieve terms suitable for displaying and another one for using on further searches [1]. The Net::Z3950::ZOOM implementation actually provides both [2] but we where using the wrong one. Using $scanset->display_term(...) instead of $scanset->term(...) fixes the problem. To test: - Do a index scan search (advanced search > more options > check 'index scan') - Notice non-latin characters are replaced by one or more '@' symbols. - Apply the patch - Re-do the search, everything shows as it should. - Try to follow any of the terms (clicking on them) and notice that it actually gives you relevant results (i.e. is not searching for @!!!!). - Sign off :-D Regards To+ [1] http://zoom.z3950.org/api/zoom-1.4.html#3.6.3 [2] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#term()_/_display_term() Sponsored-by: Universidad Nacional de Cordoba Followed test plan. Patch behaves as expected. Signed-off-by: Marc Véron <veron@veron.ch> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> I reproduced the issue and I confirm this patch fixes it. I put "Fuß" in a title, reindex the record. Launch a search on Title checking the "scan index" checkbox. And the non-latin characters are well displayed.
Pushed to master. Thanks, Tomás!