Bug 4397 - display problems (umlauts, ß) with scan index in advanced search
Summary: display problems (umlauts, ß) with scan index in advanced search
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low minor (vote)
Assignee: Tomás Cohen Arazi
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-04-22 12:31 UTC by Katrin Fischer
Modified: 2014-12-07 20:07 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
scan index with fußball (15.17 KB, image/png)
2010-04-22 12:31 UTC, Chris Cormack
Details
scan index with Währung (umlaut) (14.59 KB, image/png)
2010-04-22 12:32 UTC, Chris Cormack
Details
Norwegian trouble with scan (31.54 KB, image/png)
2010-04-22 13:17 UTC, Chris Cormack
Details
Bug 4397: Scan index search results broken for non-latin characters (1.77 KB, patch)
2014-02-28 18:35 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
[Signed-off] Bug 4397: Scan index search results broken for non-latin characters (1.96 KB, patch)
2014-03-04 15:15 UTC, Marc Véron
Details | Diff | Splinter Review
Bug 4397: Scan index search results broken for non-latin characters (2.22 KB, patch)
2014-03-20 09:48 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2010-05-21 01:26:42 UTC


---- Reported by katrin.fischer@bsz-bw.de 2010-04-22 12:31:31 ----

Created an attachment
scan index with fußball

When using scan index in advanced search with Fußball the ß is not displayed correctly. 
Fußball (soccer) -> Fu@@ball

Umlauts are reduced to the basic character (ä to a, ü to u).
Währung (currency) -> Wahrung

(see screenshots)



---- Additional Comments From katrin.fischer@bsz-bw.de 2010-04-22 12:32:39 ----

Created an attachment
scan index with Währung (umlaut)





---- Additional Comments From magnus@enger.priv.no 2010-04-22 13:17:22 ----

Created an attachment
Norwegian trouble with scan





---- Additional Comments From magnus@enger.priv.no 2010-04-22 13:18:29 ----

I see similar problems with Norwegian æøå, which give these conversions: 

æ -> @@
ø -> o
å -> a



--- Bug imported by chris@bigballofwax.co.nz 2010-05-21 01:26 UTC  ---

This bug was previously known as _bug_ 4397 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=4397
Imported an attachment (id=2068)
Imported an attachment (id=2069)
Imported an attachment (id=2070)

Actual time not defined. Setting to 0.0
Setting qa contact to the default for this product.
   This bug either had no qa contact or an invalid one.
The original submitter of attachment 2068 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 2069 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 2070 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.

Comment 1 Fridolin Somers 2013-01-09 16:54:38 UTC
Scan shows Zebra internal datas.
I think it is normal to see characters the way the are indexed, no the way they are in record.
Transformations depends on word-phrase-utf.chr config file.
Comment 2 Tomás Cohen Arazi 2014-02-28 18:23:42 UTC
I reproduced the problem and using yaz-client I noticed it was a Koha problem actually:

Z> open unix:/home/tomas/koha-dev/var/run/zebradb/bibliosocket
Connecting...OK.
Sent initrequest.
Connection accepted by v3 target.
ID     : 81
Name   : Zebra Information Server/GFS/YAZ
Version: 4.2.30 98864b44c654645bc16b2c54f822dc2e45a93031
Options: search present delSet triggerResourceCtrl scan sort extendedServices namedResultSets
Elapsed: 0.002084
Z> base biblios
Z> scan @attr 1=4 @attr 4=6 @attr 5=1 t
query: @attr 1=4 @attr 4=6 @attr 5=1 t
Received ScanResponse
20 entries, position=1
* t (60)
  tử (1)
  tạo (1)
  tập (3)
  ta (8)
  tablature (2)
  table (3)
  Tack (1)
  tacka (1)
  Tadeusz (2)
  Tage (2)
  Tagen (3)
  Taha (1)
  tài (2)
  taide (2)
  Tain (1)
  Tainted (1)
  taiteen (1)
  taiteokset (1)
  Takashi (18)
Elapsed: 0.316274
Z>


while using Koha returned the @@@ instead, as Katrina explained in the bug description. So => ASSIGNED.
Comment 3 Tomás Cohen Arazi 2014-02-28 18:35:59 UTC Comment hidden (obsolete)
Comment 4 Marc Véron 2014-03-04 15:15:14 UTC Comment hidden (obsolete)
Comment 5 Katrin Fischer 2014-03-08 20:50:33 UTC
I have problems applying this patch with git bz... 

$ git bz apply 4397
Traceback (most recent call last):
  File "/usr/sbin/git-bz", line 2609, in <module>
    do_apply(*args)
  File "/usr/sbin/git-bz", line 1688, in do_apply
    print "Bug %d - %s" % (bug.id, bug.short_desc)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xdf' in position 38: ordinal not in range(128)
Comment 6 Katrin Fischer 2014-03-08 20:51:18 UTC
But works fine using wget and git am -iu3...
Comment 7 Katrin Fischer 2014-03-14 09:32:31 UTC
Right now, I have problems reproducing the initial issue. Arabic and ß display fine for me on master without the patch, umlauts are still losing their diacritic, ü showing as u. Could it be that this works only for some specific setting?

Also if you pick "Title" from the scan search options, it explodes into a ZOOM error before and after applying the patch.

Waiting for Feedback from Tomas.
Comment 8 Katrin Fischer 2014-03-18 07:36:38 UTC
This patch doesn't break anything for me, but I have problems to replicate the original display problem on my installations now. Could someone else give this a try?
Comment 9 Jonathan Druart 2014-03-20 09:48:14 UTC
Created attachment 26470 [details] [review]
Bug 4397: Scan index search results broken for non-latin characters

The Zoom specification defines that a ScanSet should provide a way
to retrieve terms suitable for displaying and another one for using
on further searches [1].

The Net::Z3950::ZOOM implementation actually provides both [2] but we
where using the wrong one.

Using $scanset->display_term(...) instead of $scanset->term(...) fixes the problem.

To test:
- Do a index scan search (advanced search > more options > check 'index scan')
- Notice non-latin characters are replaced by one or more '@' symbols.
- Apply the patch
- Re-do the search, everything shows as it should.
- Try to follow any of the terms (clicking on them) and notice that
  it actually gives you relevant results (i.e. is not searching for @!!!!).
- Sign off :-D

Regards
To+

[1] http://zoom.z3950.org/api/zoom-1.4.html#3.6.3
[2] http://search.cpan.org/~mirk/Net-Z3950-ZOOM/lib/ZOOM.pod#term()_/_display_term()

Sponsored-by: Universidad Nacional de Cordoba

Followed test plan. Patch behaves as expected.
Signed-off-by: Marc Véron <veron@veron.ch>

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
I reproduced the issue and I confirm this patch fixes it.
I put "Fuß" in a title, reindex the record. Launch a search on Title
checking the "scan index" checkbox. And the non-latin characters are
well displayed.
Comment 10 Galen Charlton 2014-05-05 05:27:23 UTC
Pushed to master.  Thanks, Tomás!