Bug 5616 - UTF-8 problem in Card View
Summary: UTF-8 problem in Card View
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: PATCH-Sent (DO NOT USE) minor (vote)
Target Milestone: ---
Assignee: Frédérick Capovilla
QA Contact: Bugs List
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-01-13 15:44 UTC by Frédérick Capovilla
Modified: 2013-12-05 20:04 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Adds utf8::encode on the output in Card View. (392 bytes, patch)
2011-01-13 15:44 UTC, Frédérick Capovilla
Details | Diff | Splinter Review
Adds utf8::encode on the output in Card View. (Correct patch formatting) (752 bytes, patch)
2011-02-07 14:36 UTC, Frédérick Capovilla
Details | Diff | Splinter Review
Follow up patch (1.24 KB, patch)
2011-10-10 13:00 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Revised patch (1.81 KB, patch)
2011-10-12 14:11 UTC, Marcel de Rooy
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 5616: UTF-8 problem in Card View / Follow up (1.95 KB, patch)
2011-10-12 21:36 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 5616: UTF-8 problem in Card View / Follow up (1.92 KB, patch)
2011-10-18 15:39 UTC, Paul Poulain
Details | Diff | Splinter Review
fr-CA version (Pootle po files) (47.94 KB, image/png)
2011-11-04 15:07 UTC, Frédérick Capovilla
Details
En version (40.38 KB, image/png)
2011-11-04 15:07 UTC, Frédérick Capovilla
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Frédérick Capovilla 2011-01-13 15:44:47 UTC Comment hidden (obsolete)
Comment 1 Katrin Fischer 2011-01-13 22:25:15 UTC
Hi Frédérick, 

I checked the card view of z39.50 search and cataloging search using in example with German umlauts. I remember seeing a problem with diacritics in one of the views, but can not reproduce it now. Do you have an example and can tell me which view is broken?

Your attached patch misses the From, Date and Subject lines - perhaps you are missing something in your git configuration?
Comment 2 Frédérick Capovilla 2011-01-14 15:18:18 UTC
(In reply to comment #1)
> Hi Frédérick, 
> 
> I checked the card view of z39.50 search and cataloging search using in example
> with German umlauts. I remember seeing a problem with diacritics in one of the
> views, but can not reproduce it now. Do you have an example and can tell me
> which view is broken?
> 
> Your attached patch misses the From, Date and Subject lines - perhaps you are
> missing something in your git configuration?

The encoding problems were on this page :
/koha/catalogue/showmarc.pl?viewas=card&id=1

You can access it in the cataloging section ( /koha/cataloguing/addbooks.pl?q=%24b ), when you click on the "Card" link the the preview column for a search result.


I created the patch using the "git diff", so that might be why there is no "From, Date and Subject" lines. Do I need to send a new version of my patchs, using "git format-patch" ?
Comment 3 Owen Leonard 2011-01-31 16:45:57 UTC
I can confirm that diacritics are now showing correctly with Frédérick's patch applied. However, I can't do an official sign-off (and give Frédérick credit) without a properly formatted patch.
Comment 4 Frédérick Capovilla 2011-02-07 14:36:19 UTC Comment hidden (obsolete)
Comment 5 Chris Cormack 2011-02-08 08:20:26 UTC
Patched pushed, please test
Comment 6 Owen Leonard 2011-02-16 16:34:12 UTC
The OPAC version of showmarc.pl should get the same fix.
Comment 7 Frédérick Capovilla 2011-02-22 19:47:56 UTC
(In reply to comment #6)
> The OPAC version of showmarc.pl should get the same fix.

I did some tests with the opac version of showmarc.pl ( opac/opac-showmarc.pl ) and, strangely, there is no encoding problem with it.

Also, it seems the card view found in opac-showmarc.pl isn't used anywhere in Koha right now.
Comment 8 Marcel de Rooy 2011-02-23 12:21:24 UTC
(In reply to comment #7)
> (In reply to comment #6)
> > The OPAC version of showmarc.pl should get the same fix.
> I did some tests with the opac version of showmarc.pl ( opac/opac-showmarc.pl )
> and, strangely, there is no encoding problem with it.
> Also, it seems the card view found in opac-showmarc.pl isn't used anywhere in
> Koha right now.

See also comment 17 of bug 5106. The code was reformatted, but there is no call anymore to that part of card view.
Chris replied that some people might still use the code in saved URLs.
Comment 9 Marcel de Rooy 2011-10-10 10:22:48 UTC
See also bug 6996. Includes a patch for current master (3.5.0.14) fixing encoding problem in opac-showmarc.pl
Comment 10 Marcel de Rooy 2011-10-10 12:59:53 UTC
Frederick,
I saw that in your patch you encoded the data.
AFAIK you should decode data coming from outside the program and you could leave the encoding to perl when outputting the data as long as you have the encoding of your file handles correctly.
Nevertheless, I still think your patch will work in most cases.
Note that it is not needed to decode data from LibXSLT if the stylesheet contains the instruction to use UTF8 (See CPAN.)
I would suggest to add the instruction to use UTF8 in the HTML header.

Sending a small additional patch for that purpose. Could you check it?
Comment 11 Marcel de Rooy 2011-10-10 13:00:26 UTC Comment hidden (obsolete)
Comment 12 Frédérick Capovilla 2011-10-11 14:47:44 UTC
(In reply to comment #11)
> Created attachment 5826 [details] [review]
> Follow up patch
> 
> Follow up

I tested the follow up patch on my instances using Firefox, and the accented characters became "�"s again.
This new patch doesn't seem to fix the problem, not sure why.
Comment 13 Frédérick Capovilla 2011-10-11 14:57:28 UTC
Additionnal details :
If I remove the "-charset => 'UTF-8'" in the header() call of your last patch, the accented are shown correctly, but Firefox tells me the encoding of the page is ISO-8859-1.

On the old signed off patch, Firefox tells me the encoding of the page is UTF-8.
Comment 14 Marcel de Rooy 2011-10-12 11:45:17 UTC
(In reply to comment #13)
> Additionnal details :
> If I remove the "-charset => 'UTF-8'" in the header() call of your last patch,
> the accented are shown correctly, but Firefox tells me the encoding of the page
> is ISO-8859-1.
> 
> On the old signed off patch, Firefox tells me the encoding of the page is
> UTF-8.

Our goal is definitely: The page should be in UTF-8. So that should be in the header, but we have to solve the other issue of course.

I did some tests and had correct encodings (before sending the patch), although it seemed that the cache sometimes sits in between and does not show the updated page.

Frederic, are you sure (at first) that your characters are saved in UTF-8 in MySQL? Did you check diacritics in the Latin range only or also other characters (from other parts of Unicode) ?
Comment 15 Marcel de Rooy 2011-10-12 14:11:53 UTC Comment hidden (obsolete)
Comment 16 Frédérick Capovilla 2011-10-12 14:58:55 UTC
> Frederic, are you sure (at first) that your characters are saved in UTF-8 in
> MySQL? Did you check diacritics in the Latin range only or also other
> characters (from other parts of Unicode) ?

My characters are all saved in utf8 with the utf8_general_ci collation in MySQL.

Since most of our records are in french, I'm guessing I only tested the diacritics with the Latin range.

I'll test your new patch and report back.
Comment 17 Frédérick Capovilla 2011-10-12 15:27:58 UTC
(In reply to comment #15)
> Created attachment 5853 [details] [review]
> Revised patch
> 
> Frederic, could you please test this one too?
> With only Latin chars such as: Das Schöne Süden 
> And with stuff like: Awāʾil al-maṭbūʿāt al-ūrūbbiyya
> 
> Thanks!

I tested this patch and it seems to work correctly. The page is always in UTF-8 and all characters, Latin and non-Latin appear correctly.

Note that the patch in attachment 3074 [details] [review], while less complete, also gives the same results.
Comment 18 Katrin Fischer 2011-10-12 21:36:45 UTC Comment hidden (obsolete)
Comment 19 Paul Poulain 2011-10-18 15:28:43 UTC
QA comment :

Same thing as Bug 6996, except there is a trailing commented line that does not need to be here:
+#$formatted=Encode::decode_utf8($formatted) unless utf8::is_utf8($formatted);
+#seems not to be necessary

Will provide an updated fix without thit comment and mark passed QA
Comment 20 Paul Poulain 2011-10-18 15:39:39 UTC
Created attachment 5958 [details] [review]
Bug 5616: UTF-8 problem in Card View / Follow up

Decode data only if needed.
Add instruction to use UTF8 in html header.
Force utf8 on output handle.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Umlauts and other diacritics in card view of z39.50 search are now
correctly displayed.

Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
  removed 2 commented lines
Comment 21 Chris Cormack 2011-10-19 03:46:23 UTC
Pushed, please test
Comment 22 Frédérick Capovilla 2011-11-02 15:57:25 UTC
I encountered another encoding problem with this new patch :
There are still encoding problems if I try to watch the showmarc details of a record while using the fr-CA language.
What's strange is that the encoding is correct in the "en" language.

To reproduce :
- Have two languages set in your system preferences (I tried with "en" and "fr-CA").
- Go to the showmarc.pl page for a biblio record containing utf-8 characters ( cgi-bin/koha/catalogue/showmarc.pl?id=1 )
- Alternate between the two languages using the language bar at the bottom and see the difference in encoding. For me, the "fr-CA" version of the page had encoding problems and the "en" one was ok.
Comment 23 Marcel de Rooy 2011-11-03 09:23:33 UTC
(In reply to comment #22)
> I encountered another encoding problem with this new patch :
> There are still encoding problems if I try to watch the showmarc details of a
> record while using the fr-CA language.
> What's strange is that the encoding is correct in the "en" language.
> To reproduce :
> - Have two languages set in your system preferences (I tried with "en" and
> "fr-CA").
> - Go to the showmarc.pl page for a biblio record containing utf-8 characters (
> cgi-bin/koha/catalogue/showmarc.pl?id=1 )
> - Alternate between the two languages using the language bar at the bottom and
> see the difference in encoding. For me, the "fr-CA" version of the page had
> encoding problems and the "en" one was ok.

Frederic,
Could you please attach two screenshots?
I tested with English and Dutch (nl-NL), but did not have differences.
Looked at two records, one with Latin diacritics, one with non-Latin unicode.
Comment 24 Frédérick Capovilla 2011-11-04 15:06:27 UTC
> Could you please attach two screenshots?
> I tested with English and Dutch (nl-NL), but did not have differences.
> Looked at two records, one with Latin diacritics, one with non-Latin unicode.

I did more tests and it seems the problem happens in the fr-FR language, and the fr-CA from Pootle. It might be a translation problem in the po files, but I still can't find what's wrong.
Comment 25 Frédérick Capovilla 2011-11-04 15:07:12 UTC
Created attachment 6203 [details]
fr-CA version (Pootle po files)
Comment 26 Frédérick Capovilla 2011-11-04 15:07:28 UTC
Created attachment 6204 [details]
En version
Comment 27 Marcel de Rooy 2011-11-04 15:55:39 UTC
(In reply to comment #24)
> I did more tests and it seems the problem happens in the fr-FR language, and
> the fr-CA from Pootle. It might be a translation problem in the po files, but I still can't find what's wrong.

I installed fr-FR from the Koha 3.6.0 install, but do not have a display problem for Latin and non-Latin diacritics. 
Could your problem be related to bug 6895 ? You could also try a new install of fr-CA Pootle version with that additional patch?
Comment 28 Jared Camins-Esakov 2012-12-31 00:14:43 UTC
There have been no further reports of problems so I am marking this bug resolved.