Bug 12279

Summary: Diacritics in system preference editor broken
Product: Koha Reporter: Katrin Fischer <katrin.fischer>
Component: I18N/L10NAssignee: Tomás Cohen Arazi <tomascohen>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: normal    
Priority: P5 - low CC: bgkriegel, f.demians, gmcharlt, jonathan.druart, m.de.rooy, nicolas.legrand
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Screenshot of German OPAC tab in sys pref editor
Screenshot of French OPAC tab in sys pref editor
Screenshot of German Logs tab in sys pref editor
Bug 12279: Diacritics in system preference editor broken
Bug 12279: Diacritics in system preference editor broken
[SIGNED-OFF] Bug 12279: Diacritics in system preference editor broken
Bug 12279: Diacritics in system preference editor broken
Bug 12279: Diacritics in system preference editor broken
Bug 12279: Diacritics in system preference editor broken
Bug 12279: Counterpatch touching only preferences.pl
Bug 12279: Diacritics in system preference editor broken
Bug 12279: Diacritics in system preference editor broken

Description Katrin Fischer 2014-05-18 21:00:00 UTC
Created attachment 28316 [details]
Screenshot of German OPAC tab in sys pref editor

I see some strange behaviour with diacritics in the system preference editor on current master.

What I did:
- cd misc/translator
- perl translate update de-DE
- perl translate install de-DE

The strange thing is that only some pages/umlauts are broken, while others  on other tabs appear ok. I see the same behaviour with French - attaching screenshots.
Comment 1 Katrin Fischer 2014-05-18 21:00:39 UTC
Created attachment 28317 [details]
Screenshot of French OPAC tab in sys pref editor
Comment 2 Katrin Fischer 2014-05-18 21:06:05 UTC
Created attachment 28318 [details]
Screenshot of German Logs tab in sys pref editor

And while OPAC is broken, the Logs tab displays diacritics correctly.
Comment 3 Nicolas Legrand 2014-05-20 11:57:18 UTC
On a 3.15.00.052 it looks I have the same problem. I made

cd misc/translator
./translate update de-DE
./translate install de-DE
./translate update fr-FR
./translate install fr-FR
./translate update tr-TR
./translate install tr-TR

Some are broken, like opaccolorstylesheet :

fr-FR: Inclure la feuille de style CSS supplémentaire pour outrepasser les paramètres spécifiques depuis la feuille de style par défaut (laisser vide pour désactiver). Inscrire juste le nom du fichier, un chemin local complet ou une URL complète commençant par http:// (si le fichier est sur un serveur distant). Merci de noter que si seul un nom de fichier est entré, le fichier devra se trouver dans le sous-dossier css pour chaque thème et langage actif dans le dossier des modèles de Koha. Un chemin local est attendu pour démarrer depuis votre fichier HTTP source. 

de-DE: Verwende zusätzlich das CSS-Stylesheet um Anweisungen im vorhandenen Stylesheet zu überschreiben (frei lassen, wenn nicht verwendet). Geben Sie nur einen Dateinamen oder einen vollständigen lokalen Pfad oder eine vollständige URL beginnend mit http:// an (falls die Datei auf einem anderen Server liegt). Wenn nur ein Dateiname angegeben wird, sollte die Datei im Verzeichnis css für das aktive Theme und die jeweilige Sprache gespeichert sein. Ein vollständiger lokaler Pfad beginnt mit dem Wurzelverzeichnis des Webservers.

Some are OK, like language :

fr-FR: Activer à l'interface professionnelle les langues suivantes :

Deutsch(de-DE)
English(en)
Türkçe(tr-TR)
Français(fr-FR) 

de-DE: Aktiviere folgende Sprachen in der Dienstoberfläche:

Deutsch(de-DE)
Français(fr-FR)
Türkçe(tr-TR)
English(en)
Comment 4 Jonathan Druart 2014-05-20 13:07:51 UTC
I reproduce with perl v5.18. Bernardo and Fridolin don't reproduce with perl v5.14.
Comment 5 Bernardo Gonzalez Kriegel 2014-05-20 16:40:45 UTC
Please report version of YAML::Syck

Have no problems on Ubuntu 12.04 (ver 1.19), or Debian 7 (ver 1.20),
but with latest version (ver 1.27, from cpan) I can reproduce
Comment 6 Katrin Fischer 2014-05-20 17:29:32 UTC
Yaml:Syck: 1.27 - It looks like we have a candidate.
Comment 7 Tomás Cohen Arazi 2014-05-20 17:32:34 UTC Comment hidden (obsolete)
Comment 8 Bernardo Gonzalez Kriegel 2014-05-20 20:59:05 UTC Comment hidden (obsolete)
Comment 9 Bernardo Gonzalez Kriegel 2014-05-20 20:59:33 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2014-05-21 06:43:36 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2014-05-21 06:44:18 UTC
YAML::Syck is v1.27 here and I confirm this patch fixes the issue.
But I'm still wondering why the pref files contain a mix of '\xC3\xA8' and 'é'.

Marked as Passed QA.
Comment 12 Galen Charlton 2014-05-21 19:22:41 UTC
Two comments:

This causes a regression, as a few of the tabs actually use TT:

circulation.pref:            - '[% local_currency %] in fines.'
circulation.pref:            - '[% local_currency %] in fines.'
opac.pref:            - '[% local_currency %] in fines (leave blank to disable).'
patrons.pref:         - '[% local_currency %].'

Failing QA on that basis.

Also, I'm not sure YAML::Syck 1.27 is specifically to blame -- on a Debian Wheezy system, when I installed YAML::Syck 1.27 from CPAN, I did not see the bug.
Comment 13 Tomás Cohen Arazi 2014-05-22 11:56:35 UTC Comment hidden (obsolete)
Comment 14 Jonathan Druart 2014-05-22 12:56:32 UTC Comment hidden (obsolete)
Comment 15 Marcel de Rooy 2014-05-28 12:48:40 UTC Comment hidden (obsolete)
Comment 16 Marcel de Rooy 2014-05-28 12:50:58 UTC
Moving status to Failed QA in order to make a decision between both approaches.
Comment 17 Tomás Cohen Arazi 2014-05-29 15:33:15 UTC
(In reply to M. de Rooy from comment #15)
> Created attachment 28517 [details] [review] [review]
> Bug 12279: Counterpatch touching only preferences.pl
> 
> Instead of a new routine in Templates.pm that actually only decodes the
> data, I would recommend to put this temporary fix into preferences.pl.
> Bug 11944 should provide the real fix..

I first tried that same approach, but hesitated to implement it that way beacuse i wasn't sure about the effect of double encoding in some unknown scenario: i.e. we are messing with UTF-8 inside C4::Templates->output() and again on its output.

Anyway, I tried this (again) on both YAML::Syck 1.19 and 1.27 and the patch fixes the problem. If Galen agrees on Marcel's approach I'd like to push this one soon.
Comment 18 Tomás Cohen Arazi 2014-05-29 15:34:54 UTC Comment hidden (obsolete)
Comment 19 Jonathan Druart 2014-06-02 14:34:49 UTC
Created attachment 28614 [details] [review]
Bug 12279: Diacritics in system preference editor broken

Temporary fix into preferences.pl.
Bug 11944 should provide the real fix..

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 20 Tomás Cohen Arazi 2014-06-02 14:45:46 UTC
Patch pushed to the master branch.

Thanks Marcel!
Comment 21 Galen Charlton 2014-06-02 16:38:59 UTC
Pushed to 3.16.x.
Comment 22 Jonathan Druart 2014-06-09 10:46:33 UTC
(In reply to Jonathan Druart from comment #19)
> Created attachment 28614 [details] [review] [review]
> Bug 12279: Diacritics in system preference editor broken

Patch provided on bug 11944 (actually it just reverts this one).