Created attachment 15202 [details] 3 iamges with incorrect UNICODE Attaching 3 images - 2 for Cyrillic records and one or Armenian. You will notice that instead of the letter some other code is displayed and when pressing on the names from left pan (Refine your search), the search is returning null results (see arm1.png). Another interesting detail. When pressing om left pan name, the display on search window of Armenian name is displayed correctly (see arm1.png) but on the top (URL window) some of letters are displayed incorrectly. When we are just typing Armenian or Cyrillic scripts in the search window everything is working fine. So the problem is how is encoding/indexing going with the 'Refine your search' pan. Tigran
Hi Tigran, try increasing the value in the system preference FacetLabelTruncationLength. Your broken characters always show up where the system cuts of the string - so this could fix it.
Please reopen if that does not solve your problem.
Created attachment 15259 [details] Incorrect dispaly
(In reply to comment #2) > Please reopen if that does not solve your problem. (In reply to comment #0) > Created attachment 15202 [details] > 3 iamges with incorrect UNICODE > > Attaching 3 images - 2 for Cyrillic records and one or Armenian. > You will notice that instead of the letter some other code is displayed > and when pressing on the names from left pan (Refine your search), the > search is returning null results (see arm1.png). > Another interesting detail. > When pressing om left pan name, the display on search window of Armenian > name is displayed correctly (see arm1.png) but on the top (URL window) > some of letters are displayed incorrectly. > When we are just typing Armenian or Cyrillic scripts in the search window > everything is working fine. > So the problem is how is encoding/indexing going with the 'Refine your > search' pan. > Tigran (In reply to comment #1) > Hi Tigran, > > try increasing the value in the system preference > FacetLabelTruncationLength. Your broken characters always show up where the > system cuts of the string - so this could fix it. Hi Katrin After changes in system preference the problem is not solved (we have done reindexing). Attaching images wirh explanations. Letters are displayed in not correct way in the URL, although in the left pan the display is normal. Tgran
Hi Tigran, not sure what to do about this, but reopening the bug.
Noticed that for Cyrillic 2 capital letters are displayed incorrectly (the letters with the codes 041F and 0421). and for Armenian 2 small letters are displayed incorrectly (the letters with the cods 0561 and 0576) Tigran
Created attachment 16942 [details] [review] Proposed patch for facet labels See commit message
Hi Fridolyn, Thanks a lot for your efforts ^_^ I just have a basic information about git so I've tried to apply the changes manually and installed this from cpan TruncateByWord, see http://search.cpan.org/~kawabata/Template-Plugin-TruncateByWord-0.11/ but the results is still not good maybe I miss something :( can you help me to test this with you ? some links to read about applying a patch with git . and I wish you test it with the attached Arabic records or any one if want to test .file name : arabicrecords.mrc Many thanks for you Karam.
Created attachment 16953 [details] 23 Arabic marc record to test the changes
We have the same problem in Rennes 2, but never noticed before... M. Saby
Fridolyn, your patch does not fix the problem. Mathieu
I see the good author name in facets, but name included in url is broken name in facets حمو, محمد آيت name in record حمو--محمد آيت link in facets (not working) http://msaby-virtualbox:8080/cgi-bin/koha/catalogue/search.pl?idx=kw&q=islam&sort_by=relevance_dsc&limit=au:%20%EF%BF%BD%D9%85%D9%88,%20%D9%85%20%EF%BF%BD%D9%85%20%EF%BF%BD%20%20%EF%BF%BD%D9%8A%20%EF%BF%BD link in record (working) http://msaby-virtualbox:8080/cgi-bin/koha/catalogue/search.pl?q=au:a05%20and%20au:%D8%AD%D9%85%D9%88%20and%20au:%D9%85%D8%AD%D9%85%D8%AF%20%D8%A2%D9%8A%D8%AA M. Saby
Hi, could you please test Bug 6554. I found that that bug solves this problem.
The patch in http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554 was reverted and this problem is come back again to 3.12 .
Hi, this is no more in current master, fixed by Bug 10176. I did test using Karam records, for me facets are working. Could someone else test?
(In reply to comment #15) > Hi, > this is no more in current master, fixed by Bug 10176. > I did test using Karam records, for me facets are working. > Could someone else test? my Koha version is 3.12.00 and the problem exist. See the ? mark for Cyrillic and Armenian scripts. Tigran Cyrillic script Topics Астрофизик... Вселенная Солнечная �... Armenian script Series Գյուղական �... Ժողովրդակա... Նյութեր Հա�... Authors Գօգօլ Խաչատրյան, ... Հովնանյան, ... Պապոյան, Ել... Ռոզանով, Սե... Show more Places Շաղատ գյու�... Topics Գեղարվեստա... Հայ գրական�... Հայ ժողովր�... Ռուս գրակա�... Ռուս մանկա�...
Hi Tigran, you can have two problems: wrong display and wrong links, 10176 fixes the link, not the display :-) To fix the display we need to rewrite how facets are trimmed, at this moment it use preference FacetLabelTruncationLength, that truncates facet text at that lenght, which is alright but only for scripts that use latin or not ligate chars. It needs to be rewritten so it cut facet text on " " (space) position <= FacetLabelTruncationLength. A workaround is set FacetLabelTruncationLength to a bigger value, 50 for example, then facets will not show those <?>. The test I was talking about involves the links :-)
Created attachment 24425 [details] [review] Bug 9579: use String::Multibyte to correctly truncate facets Bug 9579: use String::Multibyte to correctly truncate facets As Bernardo pointed on comment #17, the incorrect display of multibyte strings in facets is related to the way they are currently truncated. C4::Search::getRecords relies on CORE::substr for the task, which only counts bytes, and thus doesn't check it is breaking multi-byte characters. This patch replaces both the use of CORE::substr and CORE::length for their String::Multibyte counterpart. To test: - Set FacetLabelTruncationLength = 4 to make sure it will try to truncate the facet label. - Add a record with multiple multibyte characters in the author name (copy and paste 'ակ' several times so it is looong). Note: make sure there are at least two records so you're presented the search results. - Reindex zebra to make it possible to search the record. - Search for the record. - Notice the facet with cyrillic characters is broken in the place it gets truncated. - Apply the patch and repeat the search. - Notice the string is not broken. - Sign off. Regards To+
Created attachment 24426 [details] [review] Bug 9579: (followup) add dependency for String::Multibyte Bug 9579: (followup) add dependency for String::Multibyte
Created attachment 24430 [details] [review] Bug 9579: Facets truncation broken for multi-byte characters We seem to be relying on whatever Zoom::Results->render return, and Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr (and probably CORE::length too) cut the bytes wrong. This patch just decodes the UTF-8 data that render() returns and then Perl behaves, heh. It uses Encode::decode_utf8 which is already a dependency for the current stable Koha releases. To test: - Set FacetLabelTruncationLength = 4 to make sure it will try to truncate the facet label. - Add a record with multiple multibyte characters in the author name (copy and paste a cyrillic character several times so it is looong). Note: make sure there are at least two records so you're presented the search results. - Reindex zebra to make it possible to search the record. - Search for the record. - Notice the facet with cyrillic characters is broken in the place it gets truncated. - Apply the patch and repeat the search. - Notice the string is not broken. - Sign off. Regards To+ Sponsored-by: Universidad Nacional de Cordoba
Sorry for spamming this bug :-D This solution should be used for the stable branches. The work on bug 11096 already covers this problem as the whole record fields are decoded in the first place. I hope this can move fast so it is on stable soon, as backporting 11096 seems a bit more difficult.
Created attachment 24684 [details] [review] Facets truncation broken for multi-byte characters You may have issues trying to git bz this. I had to upload this old-school, because I have يكيمكتبات : قبسي ، كرم in the commit message.
REVISED TEST PLAN ----------------- 1) Import the attached sample records. 2) Rebuild your indexes 3) In OPAC search for ﻲﻜﻴﻤﻜﺘﺑﺎﺗ : ﻖﺒﺴﻳ ، ﻙﺮﻣ -- There will be ugly diamonds with question marks in the facets 4) apply the patch 5) Search again. -- The names will be properly truncated. NOTE: This test assumes FacetLabelTruncationLength = 20.
Created attachment 24725 [details] [review] [PASSED QA] Bug 9579: Facets truncation broken for multi-byte characters We seem to be relying on whatever Zoom::Results->render return, and Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr (and probably CORE::length too) cut the bytes wrong. This patch just decodes the UTF-8 data that render() returns and then Perl behaves, heh. It uses Encode::decode_utf8 which is already a dependency for the current stable Koha releases. REVISED TEST PLAN ----------------- 1) Import the attached sample records. 2) Rebuild your indexes 3) In OPAC search for يكيمكتبات : قبسي ، كرم -- There will be ugly diamonds with question marks in the facets 4) apply the patch 5) Search again. -- The names will be properly truncated. NOTE: This test assumes FacetLabelTruncationLength = 20. Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes all tests and QA script. Works as described, tested with several German, English and the Arabic test record. Arabic strings now display correctly and no regression was found.
Created attachment 24959 [details] [review] Bug 9579: fix truncation of facets containing multi-byte characters We seem to be relying on whatever Zoom::Results->render return, and Perl doesn't explicitly consider it UNICODE data. That's why CORE::substr (and probably CORE::length too) cut the bytes wrong. This patch just decodes the UTF-8 data that render() returns and then Perl behaves, heh. It uses Encode::decode_utf8 which is already a dependency for the current stable Koha releases. REVISED TEST PLAN ----------------- 1) Import the attached sample records. 2) Rebuild your indexes 3) In OPAC search for يكيمكتبات : قبسي ، كرم -- There will be ugly diamonds with question marks in the facets 4) apply the patch 5) Search again. -- The names will be properly truncated. NOTE: This test assumes FacetLabelTruncationLength = 20. Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes all tests and QA script. Works as described, tested with several German, English and the Arabic test record. Arabic strings now display correctly and no regression was found. Signed-off-by: Galen Charlton <gmc@esilibrary.com> I've reviewed it and approve its inclusion in 3.14.x and earlier. I will use the patches for bug 11096, once they pass QA, for the master branch. Signed-off-by: Galen Charlton <gmc@esilibrary.com>
Patch pushed to 3.14.x, will be in 3.14.4. RM announced this bug can be pushed in stable branches but will not be in master, waiting for Bug 11096.
This patch has been pushed to 3.12.x, will be in 3.12.11.