Created attachment 2987 [details] [review] Adds utf8::encode on the output in Card View. Special characters are show as � when showing the informations of a record in Card View. Patch attached. Adds a utf8::encode on the output.
Hi Frédérick, I checked the card view of z39.50 search and cataloging search using in example with German umlauts. I remember seeing a problem with diacritics in one of the views, but can not reproduce it now. Do you have an example and can tell me which view is broken? Your attached patch misses the From, Date and Subject lines - perhaps you are missing something in your git configuration?
(In reply to comment #1) > Hi Frédérick, > > I checked the card view of z39.50 search and cataloging search using in example > with German umlauts. I remember seeing a problem with diacritics in one of the > views, but can not reproduce it now. Do you have an example and can tell me > which view is broken? > > Your attached patch misses the From, Date and Subject lines - perhaps you are > missing something in your git configuration? The encoding problems were on this page : /koha/catalogue/showmarc.pl?viewas=card&id=1 You can access it in the cataloging section ( /koha/cataloguing/addbooks.pl?q=%24b ), when you click on the "Card" link the the preview column for a search result. I created the patch using the "git diff", so that might be why there is no "From, Date and Subject" lines. Do I need to send a new version of my patchs, using "git format-patch" ?
I can confirm that diacritics are now showing correctly with Frédérick's patch applied. However, I can't do an official sign-off (and give Frédérick credit) without a properly formatted patch.
Created attachment 3074 [details] [review] Adds utf8::encode on the output in Card View. (Correct patch formatting) Patch corrected. Sorry for the inconvenience.
Patched pushed, please test
The OPAC version of showmarc.pl should get the same fix.
(In reply to comment #6) > The OPAC version of showmarc.pl should get the same fix. I did some tests with the opac version of showmarc.pl ( opac/opac-showmarc.pl ) and, strangely, there is no encoding problem with it. Also, it seems the card view found in opac-showmarc.pl isn't used anywhere in Koha right now.
(In reply to comment #7) > (In reply to comment #6) > > The OPAC version of showmarc.pl should get the same fix. > I did some tests with the opac version of showmarc.pl ( opac/opac-showmarc.pl ) > and, strangely, there is no encoding problem with it. > Also, it seems the card view found in opac-showmarc.pl isn't used anywhere in > Koha right now. See also comment 17 of bug 5106. The code was reformatted, but there is no call anymore to that part of card view. Chris replied that some people might still use the code in saved URLs.
See also bug 6996. Includes a patch for current master (3.5.0.14) fixing encoding problem in opac-showmarc.pl
Frederick, I saw that in your patch you encoded the data. AFAIK you should decode data coming from outside the program and you could leave the encoding to perl when outputting the data as long as you have the encoding of your file handles correctly. Nevertheless, I still think your patch will work in most cases. Note that it is not needed to decode data from LibXSLT if the stylesheet contains the instruction to use UTF8 (See CPAN.) I would suggest to add the instruction to use UTF8 in the HTML header. Sending a small additional patch for that purpose. Could you check it?
Created attachment 5826 [details] [review] Follow up patch Follow up
(In reply to comment #11) > Created attachment 5826 [details] [review] > Follow up patch > > Follow up I tested the follow up patch on my instances using Firefox, and the accented characters became "�"s again. This new patch doesn't seem to fix the problem, not sure why.
Additionnal details : If I remove the "-charset => 'UTF-8'" in the header() call of your last patch, the accented are shown correctly, but Firefox tells me the encoding of the page is ISO-8859-1. On the old signed off patch, Firefox tells me the encoding of the page is UTF-8.
(In reply to comment #13) > Additionnal details : > If I remove the "-charset => 'UTF-8'" in the header() call of your last patch, > the accented are shown correctly, but Firefox tells me the encoding of the page > is ISO-8859-1. > > On the old signed off patch, Firefox tells me the encoding of the page is > UTF-8. Our goal is definitely: The page should be in UTF-8. So that should be in the header, but we have to solve the other issue of course. I did some tests and had correct encodings (before sending the patch), although it seemed that the cache sometimes sits in between and does not show the updated page. Frederic, are you sure (at first) that your characters are saved in UTF-8 in MySQL? Did you check diacritics in the Latin range only or also other characters (from other parts of Unicode) ?
Created attachment 5853 [details] [review] Revised patch Frederic, could you please test this one too? With only Latin chars such as: Das Schöne Süden And with stuff like: Awāʾil al-maṭbūʿāt al-ūrūbbiyya Thanks!
> Frederic, are you sure (at first) that your characters are saved in UTF-8 in > MySQL? Did you check diacritics in the Latin range only or also other > characters (from other parts of Unicode) ? My characters are all saved in utf8 with the utf8_general_ci collation in MySQL. Since most of our records are in french, I'm guessing I only tested the diacritics with the Latin range. I'll test your new patch and report back.
(In reply to comment #15) > Created attachment 5853 [details] [review] > Revised patch > > Frederic, could you please test this one too? > With only Latin chars such as: Das Schöne Süden > And with stuff like: Awāʾil al-maṭbūʿāt al-ūrūbbiyya > > Thanks! I tested this patch and it seems to work correctly. The page is always in UTF-8 and all characters, Latin and non-Latin appear correctly. Note that the patch in attachment 3074 [details] [review], while less complete, also gives the same results.
Created attachment 5866 [details] [review] [SIGNED-OFF] Bug 5616: UTF-8 problem in Card View / Follow up Decode data only if needed. Add instruction to use UTF8 in html header. Force utf8 on output handle. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Umlauts and other diacritics in card view of z39.50 search are now correctly displayed.
QA comment : Same thing as Bug 6996, except there is a trailing commented line that does not need to be here: +#$formatted=Encode::decode_utf8($formatted) unless utf8::is_utf8($formatted); +#seems not to be necessary Will provide an updated fix without thit comment and mark passed QA
Created attachment 5958 [details] [review] Bug 5616: UTF-8 problem in Card View / Follow up Decode data only if needed. Add instruction to use UTF8 in html header. Force utf8 on output handle. Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Umlauts and other diacritics in card view of z39.50 search are now correctly displayed. Signed-off-by: Paul Poulain <paul.poulain@biblibre.com> removed 2 commented lines
Pushed, please test
I encountered another encoding problem with this new patch : There are still encoding problems if I try to watch the showmarc details of a record while using the fr-CA language. What's strange is that the encoding is correct in the "en" language. To reproduce : - Have two languages set in your system preferences (I tried with "en" and "fr-CA"). - Go to the showmarc.pl page for a biblio record containing utf-8 characters ( cgi-bin/koha/catalogue/showmarc.pl?id=1 ) - Alternate between the two languages using the language bar at the bottom and see the difference in encoding. For me, the "fr-CA" version of the page had encoding problems and the "en" one was ok.
(In reply to comment #22) > I encountered another encoding problem with this new patch : > There are still encoding problems if I try to watch the showmarc details of a > record while using the fr-CA language. > What's strange is that the encoding is correct in the "en" language. > To reproduce : > - Have two languages set in your system preferences (I tried with "en" and > "fr-CA"). > - Go to the showmarc.pl page for a biblio record containing utf-8 characters ( > cgi-bin/koha/catalogue/showmarc.pl?id=1 ) > - Alternate between the two languages using the language bar at the bottom and > see the difference in encoding. For me, the "fr-CA" version of the page had > encoding problems and the "en" one was ok. Frederic, Could you please attach two screenshots? I tested with English and Dutch (nl-NL), but did not have differences. Looked at two records, one with Latin diacritics, one with non-Latin unicode.
> Could you please attach two screenshots? > I tested with English and Dutch (nl-NL), but did not have differences. > Looked at two records, one with Latin diacritics, one with non-Latin unicode. I did more tests and it seems the problem happens in the fr-FR language, and the fr-CA from Pootle. It might be a translation problem in the po files, but I still can't find what's wrong.
Created attachment 6203 [details] fr-CA version (Pootle po files)
Created attachment 6204 [details] En version
(In reply to comment #24) > I did more tests and it seems the problem happens in the fr-FR language, and > the fr-CA from Pootle. It might be a translation problem in the po files, but I still can't find what's wrong. I installed fr-FR from the Koha 3.6.0 install, but do not have a display problem for Latin and non-Latin diacritics. Could your problem be related to bug 6895 ? You could also try a new install of fr-CA Pootle version with that additional patch?
There have been no further reports of problems so I am marking this bug resolved.