Bug 9579 - Facets truncation broken for multi-byte characters
Summary: Facets truncation broken for multi-byte characters
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: 3.14
Hardware: All All
: P1 - high major (vote)
Assignee: Tomás Cohen Arazi
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-09 13:52 UTC by Tigran
Modified: 2015-06-04 23:23 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
3 iamges with incorrect UNICODE (185.00 KB, application/msword)
2013-02-09 13:52 UTC, Tigran
Details
Incorrect dispaly (162.04 KB, application/pdf)
2013-02-11 19:45 UTC, Tigran
Details
Proposed patch for facet labels (9.66 KB, patch)
2013-03-27 15:51 UTC, Fridolin Somers
Details | Diff | Splinter Review
23 Arabic marc record to test the changes (13.56 KB, application/octet-stream)
2013-03-27 21:33 UTC, Karam Qubsi
Details
Bug 9579: use String::Multibyte to correctly truncate facets (3.01 KB, patch)
2014-01-15 19:58 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 9579: (followup) add dependency for String::Multibyte (836 bytes, patch)
2014-01-15 19:59 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 9579: Facets truncation broken for multi-byte characters (2.27 KB, patch)
2014-01-16 00:13 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Facets truncation broken for multi-byte characters (2.15 KB, patch)
2014-01-24 13:42 UTC, Mark Tompsett
Details | Diff | Splinter Review
[PASSED QA] Bug 9579: Facets truncation broken for multi-byte characters (2.38 KB, patch)
2014-01-25 08:38 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 9579: fix truncation of facets containing multi-byte characters (2.61 KB, patch)
2014-01-31 16:41 UTC, Galen Charlton
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Tigran 2013-02-09 13:52:51 UTC Comment hidden (obsolete)
Comment 1 Katrin Fischer 2013-02-10 21:25:30 UTC
Hi Tigran,

try increasing the value in the system preference FacetLabelTruncationLength. Your broken characters always show up where the system cuts of the string - so this could fix it.
Comment 2 Katrin Fischer 2013-02-10 21:26:02 UTC
Please reopen if that does not solve your problem.
Comment 3 Tigran 2013-02-11 19:45:36 UTC Comment hidden (obsolete)
Comment 4 Tigran 2013-02-11 19:50:47 UTC
(In reply to comment #2)
> Please reopen if that does not solve your problem.

(In reply to comment #0)
> Created attachment 15202 [details]
> 3 iamges with incorrect UNICODE
> 
> Attaching 3 images - 2 for Cyrillic records and one or Armenian.
> You will notice that instead of the letter some other code is displayed
> and when pressing on the names from left pan (Refine your search), the
> search is returning null results (see arm1.png).
> Another interesting detail.
> When pressing om left pan name, the display on search window of Armenian
> name is displayed correctly (see arm1.png) but on the top (URL window)
> some of letters are displayed incorrectly.
> When we are just typing Armenian or Cyrillic scripts in the search window
> everything is working fine.
> So the problem is how is encoding/indexing going with the 'Refine your
> search' pan.
> Tigran

(In reply to comment #1)
> Hi Tigran,
> 
> try increasing the value in the system preference
> FacetLabelTruncationLength. Your broken characters always show up where the
> system cuts of the string - so this could fix it.

Hi Katrin
After changes in system preference the problem is not solved (we have done reindexing).
Attaching images wirh explanations. Letters are displayed in not correct way in  the URL, although in the left pan the display is normal.
Tgran
Comment 5 Katrin Fischer 2013-02-12 08:12:18 UTC
Hi Tigran,

not sure what to do about this, but reopening the bug.
Comment 6 Tigran 2013-02-17 19:05:07 UTC
Noticed that for Cyrillic 2 capital letters are displayed incorrectly (the letters with the codes 041F and 0421).
and for Armenian 2 small letters are displayed incorrectly (the letters with the cods 0561 and 0576)

Tigran
Comment 7 Fridolin Somers 2013-03-27 15:51:12 UTC Comment hidden (obsolete)
Comment 8 Karam Qubsi 2013-03-27 21:30:34 UTC
Hi Fridolyn,
Thanks a lot for your efforts ^_^
I just have a basic information about git 
so I've tried to apply the changes manually
and installed this from cpan  TruncateByWord, see http://search.cpan.org/~kawabata/Template-Plugin-TruncateByWord-0.11/

but the results is still not good 

maybe I miss something :(

can you help me to test this with you ? 
some links to read about applying a patch with git .

and I wish you test it with the attached Arabic records or any one if want to test .file name : arabicrecords.mrc

Many thanks for you

Karam.
Comment 9 Karam Qubsi 2013-03-27 21:33:05 UTC
Created attachment 16953 [details]
23 Arabic marc record to test the changes
Comment 10 Mathieu Saby 2013-03-27 22:03:02 UTC
We have the same problem in Rennes 2, but never noticed before...

M. Saby
Comment 11 Mathieu Saby 2013-03-27 22:24:59 UTC
Fridolyn, your patch does not fix the problem.
Mathieu
Comment 12 Mathieu Saby 2013-03-27 22:34:51 UTC
I see the good author name in facets, but name included in url is broken
name in facets
حمو, محمد آيت

name in record
حمو--محمد آيت

link in facets (not working)
http://msaby-virtualbox:8080/cgi-bin/koha/catalogue/search.pl?idx=kw&q=islam&sort_by=relevance_dsc&limit=au:%20%EF%BF%BD%D9%85%D9%88,%20%D9%85%20%EF%BF%BD%D9%85%20%EF%BF%BD%20%20%EF%BF%BD%D9%8A%20%EF%BF%BD

link in record (working)
http://msaby-virtualbox:8080/cgi-bin/koha/catalogue/search.pl?q=au:a05%20and%20au:%D8%AD%D9%85%D9%88%20and%20au:%D9%85%D8%AD%D9%85%D8%AF%20%D8%A2%D9%8A%D8%AA

M. Saby
Comment 13 Bernardo Gonzalez Kriegel 2013-03-28 17:25:45 UTC
Hi, could you please test Bug 6554. 
I found that that bug solves this problem.
Comment 14 Karam Qubsi 2013-04-27 12:04:41 UTC
The patch in http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554
was reverted and this problem is come back again to 3.12 .
Comment 15 Bernardo Gonzalez Kriegel 2013-06-05 11:01:40 UTC
Hi,
this is no more in current master, fixed by Bug 10176.
I did test using Karam records, for me facets are working.
Could someone else test?
Comment 16 Tigran 2013-06-05 11:16:44 UTC
(In reply to comment #15)
> Hi,
> this is no more in current master, fixed by Bug 10176.
> I did test using Karam records, for me facets are working.
> Could someone else test?

my Koha version is 3.12.00 and the problem exist.
See the ? mark for Cyrillic and Armenian scripts.
Tigran

Cyrillic script
Topics
    Астрофизик...
    Вселенная
    Солнечная �...

Armenian script
     Series
    Գյուղական �...
    Ժողովրդակա...
    Նյութեր Հա�...

Authors
    Գօգօլ
    Խաչատրյան, ...
    Հովնանյան, ...
    Պապոյան, Ել...
    Ռոզանով, Սե...
    Show more
Places
    Շաղատ գյու�...
Topics
    Գեղարվեստա...
    Հայ գրական�...
    Հայ ժողովր�...
    Ռուս գրակա�...
    Ռուս մանկա�...
Comment 17 Bernardo Gonzalez Kriegel 2013-06-05 11:49:52 UTC
Hi Tigran,
you can have two problems: wrong display and wrong links,
10176 fixes the link, not the display :-)

To fix the display we need to rewrite how facets are trimmed,
at this moment it use preference FacetLabelTruncationLength,
that truncates facet text at that lenght, which is alright 
but only for scripts that use latin or not ligate chars.
It needs to be rewritten so it cut facet text on " " (space)
position <= FacetLabelTruncationLength.

A workaround is set FacetLabelTruncationLength to a bigger 
value, 50 for example, then facets will not show those <?>.

The test I was talking about involves the links :-)
Comment 18 Tomás Cohen Arazi 2014-01-15 19:58:48 UTC Comment hidden (obsolete)
Comment 19 Tomás Cohen Arazi 2014-01-15 19:59:32 UTC Comment hidden (obsolete)
Comment 20 Tomás Cohen Arazi 2014-01-16 00:13:28 UTC Comment hidden (obsolete)
Comment 21 Tomás Cohen Arazi 2014-01-16 00:16:51 UTC
Sorry for spamming this bug :-D

This solution should be used for the stable branches. The work on bug 11096 already covers this problem as the whole record fields are decoded in the first place.

I hope this can move fast so it is on stable soon, as backporting 11096 seems a bit more difficult.
Comment 22 Mark Tompsett 2014-01-24 13:42:03 UTC Comment hidden (obsolete)
Comment 23 Mark Tompsett 2014-01-24 13:45:16 UTC
REVISED TEST PLAN
-----------------
1) Import the attached sample records.
2) Rebuild your indexes
3) In OPAC search for ﻲﻜﻴﻤﻜﺘﺑﺎﺗ : ﻖﺒﺴﻳ ، ﻙﺮﻣ
-- There will be ugly diamonds with question marks in the facets
4) apply the patch
5) Search again.
-- The names will be properly truncated.
NOTE: This test assumes FacetLabelTruncationLength = 20.
Comment 24 Katrin Fischer 2014-01-25 08:38:41 UTC Comment hidden (obsolete)
Comment 25 Galen Charlton 2014-01-31 16:41:34 UTC
Created attachment 24959 [details] [review]
Bug 9579: fix truncation of facets containing multi-byte characters

We seem to be relying on whatever Zoom::Results->render return, and
Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr
(and probably CORE::length too) cut the bytes wrong.

This patch just decodes the UTF-8 data that render() returns and then
Perl behaves, heh.

It uses Encode::decode_utf8 which is already a dependency for the current
stable Koha releases.

REVISED TEST PLAN
-----------------
1) Import the attached sample records.
2) Rebuild your indexes
3) In OPAC search for يكيمكتبات : قبسي ، كرم
-- There will be ugly diamonds with question marks in the facets
4) apply the patch
5) Search again.
-- The names will be properly truncated.
NOTE: This test assumes FacetLabelTruncationLength = 20.

Sponsored-by: Universidad Nacional de Cordoba

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes all tests and QA script.
Works as described, tested with several German, English and
the Arabic test record. Arabic strings now display correctly
and no regression was found.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>

I've reviewed it and approve its inclusion in 3.14.x and earlier.  I
will use the patches for bug 11096, once they pass QA, for the master
branch.

Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Comment 26 Fridolin Somers 2014-02-07 08:09:35 UTC
Patch pushed to 3.14.x, will be in 3.14.4.

RM announced this bug can be pushed in stable branches but will not be in master, waiting for Bug 11096.
Comment 27 Tomás Cohen Arazi 2014-02-07 17:10:18 UTC
This patch has been pushed to 3.12.x, will be in 3.12.11.