Bug 12279 - Diacritics in system preference editor broken
Summary: Diacritics in system preference editor broken
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Tomás Cohen Arazi
QA Contact: Marcel de Rooy
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-05-18 21:00 UTC by Katrin Fischer
Modified: 2015-06-04 23:30 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Screenshot of German OPAC tab in sys pref editor (124.59 KB, image/png)
2014-05-18 21:00 UTC, Katrin Fischer
Details
Screenshot of French OPAC tab in sys pref editor (127.03 KB, image/png)
2014-05-18 21:00 UTC, Katrin Fischer
Details
Screenshot of German Logs tab in sys pref editor (86.41 KB, image/png)
2014-05-18 21:06 UTC, Katrin Fischer
Details
Bug 12279: Diacritics in system preference editor broken (1.55 KB, patch)
2014-05-20 17:32 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 12279: Diacritics in system preference editor broken (1.92 KB, patch)
2014-05-20 20:59 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 12279: Diacritics in system preference editor broken (1.93 KB, patch)
2014-05-20 20:59 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 12279: Diacritics in system preference editor broken (1.98 KB, patch)
2014-05-21 06:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 12279: Diacritics in system preference editor broken (2.13 KB, patch)
2014-05-22 11:56 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 12279: Diacritics in system preference editor broken (2.20 KB, patch)
2014-05-22 12:56 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 12279: Counterpatch touching only preferences.pl (1.11 KB, patch)
2014-05-28 12:48 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 12279: Diacritics in system preference editor broken (1.02 KB, patch)
2014-05-29 15:34 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 12279: Diacritics in system preference editor broken (1.08 KB, patch)
2014-06-02 14:34 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2014-05-18 21:00:00 UTC
Created attachment 28316 [details]
Screenshot of German OPAC tab in sys pref editor

I see some strange behaviour with diacritics in the system preference editor on current master.

What I did:
- cd misc/translator
- perl translate update de-DE
- perl translate install de-DE

The strange thing is that only some pages/umlauts are broken, while others  on other tabs appear ok. I see the same behaviour with French - attaching screenshots.
Comment 1 Katrin Fischer 2014-05-18 21:00:39 UTC
Created attachment 28317 [details]
Screenshot of French OPAC tab in sys pref editor
Comment 2 Katrin Fischer 2014-05-18 21:06:05 UTC
Created attachment 28318 [details]
Screenshot of German Logs tab in sys pref editor

And while OPAC is broken, the Logs tab displays diacritics correctly.
Comment 3 Nicolas Legrand 2014-05-20 11:57:18 UTC
On a 3.15.00.052 it looks I have the same problem. I made

cd misc/translator
./translate update de-DE
./translate install de-DE
./translate update fr-FR
./translate install fr-FR
./translate update tr-TR
./translate install tr-TR

Some are broken, like opaccolorstylesheet :

fr-FR: Inclure la feuille de style CSS supplémentaire pour outrepasser les paramètres spécifiques depuis la feuille de style par défaut (laisser vide pour désactiver). Inscrire juste le nom du fichier, un chemin local complet ou une URL complète commençant par http:// (si le fichier est sur un serveur distant). Merci de noter que si seul un nom de fichier est entré, le fichier devra se trouver dans le sous-dossier css pour chaque thème et langage actif dans le dossier des modèles de Koha. Un chemin local est attendu pour démarrer depuis votre fichier HTTP source. 

de-DE: Verwende zusätzlich das CSS-Stylesheet um Anweisungen im vorhandenen Stylesheet zu überschreiben (frei lassen, wenn nicht verwendet). Geben Sie nur einen Dateinamen oder einen vollständigen lokalen Pfad oder eine vollständige URL beginnend mit http:// an (falls die Datei auf einem anderen Server liegt). Wenn nur ein Dateiname angegeben wird, sollte die Datei im Verzeichnis css für das aktive Theme und die jeweilige Sprache gespeichert sein. Ein vollständiger lokaler Pfad beginnt mit dem Wurzelverzeichnis des Webservers.

Some are OK, like language :

fr-FR: Activer à l'interface professionnelle les langues suivantes :

Deutsch(de-DE)
English(en)
Türkçe(tr-TR)
Français(fr-FR) 

de-DE: Aktiviere folgende Sprachen in der Dienstoberfläche:

Deutsch(de-DE)
Français(fr-FR)
Türkçe(tr-TR)
English(en)
Comment 4 Jonathan Druart 2014-05-20 13:07:51 UTC
I reproduce with perl v5.18. Bernardo and Fridolin don't reproduce with perl v5.14.
Comment 5 Bernardo Gonzalez Kriegel 2014-05-20 16:40:45 UTC
Please report version of YAML::Syck

Have no problems on Ubuntu 12.04 (ver 1.19), or Debian 7 (ver 1.20),
but with latest version (ver 1.27, from cpan) I can reproduce
Comment 6 Katrin Fischer 2014-05-20 17:29:32 UTC
Yaml:Syck: 1.27 - It looks like we have a candidate.
Comment 7 Tomás Cohen Arazi 2014-05-20 17:32:34 UTC Comment hidden (obsolete)
Comment 8 Bernardo Gonzalez Kriegel 2014-05-20 20:59:05 UTC Comment hidden (obsolete)
Comment 9 Bernardo Gonzalez Kriegel 2014-05-20 20:59:33 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2014-05-21 06:43:36 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2014-05-21 06:44:18 UTC
YAML::Syck is v1.27 here and I confirm this patch fixes the issue.
But I'm still wondering why the pref files contain a mix of '\xC3\xA8' and 'é'.

Marked as Passed QA.
Comment 12 Galen Charlton 2014-05-21 19:22:41 UTC
Two comments:

This causes a regression, as a few of the tabs actually use TT:

circulation.pref:            - '[% local_currency %] in fines.'
circulation.pref:            - '[% local_currency %] in fines.'
opac.pref:            - '[% local_currency %] in fines (leave blank to disable).'
patrons.pref:         - '[% local_currency %].'

Failing QA on that basis.

Also, I'm not sure YAML::Syck 1.27 is specifically to blame -- on a Debian Wheezy system, when I installed YAML::Syck 1.27 from CPAN, I did not see the bug.
Comment 13 Tomás Cohen Arazi 2014-05-22 11:56:35 UTC Comment hidden (obsolete)
Comment 14 Jonathan Druart 2014-05-22 12:56:32 UTC Comment hidden (obsolete)
Comment 15 Marcel de Rooy 2014-05-28 12:48:40 UTC Comment hidden (obsolete)
Comment 16 Marcel de Rooy 2014-05-28 12:50:58 UTC
Moving status to Failed QA in order to make a decision between both approaches.
Comment 17 Tomás Cohen Arazi 2014-05-29 15:33:15 UTC
(In reply to M. de Rooy from comment #15)
> Created attachment 28517 [details] [review] [review]
> Bug 12279: Counterpatch touching only preferences.pl
> 
> Instead of a new routine in Templates.pm that actually only decodes the
> data, I would recommend to put this temporary fix into preferences.pl.
> Bug 11944 should provide the real fix..

I first tried that same approach, but hesitated to implement it that way beacuse i wasn't sure about the effect of double encoding in some unknown scenario: i.e. we are messing with UTF-8 inside C4::Templates->output() and again on its output.

Anyway, I tried this (again) on both YAML::Syck 1.19 and 1.27 and the patch fixes the problem. If Galen agrees on Marcel's approach I'd like to push this one soon.
Comment 18 Tomás Cohen Arazi 2014-05-29 15:34:54 UTC Comment hidden (obsolete)
Comment 19 Jonathan Druart 2014-06-02 14:34:49 UTC
Created attachment 28614 [details] [review]
Bug 12279: Diacritics in system preference editor broken

Temporary fix into preferences.pl.
Bug 11944 should provide the real fix..

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Comment 20 Tomás Cohen Arazi 2014-06-02 14:45:46 UTC
Patch pushed to the master branch.

Thanks Marcel!
Comment 21 Galen Charlton 2014-06-02 16:38:59 UTC
Pushed to 3.16.x.
Comment 22 Jonathan Druart 2014-06-09 10:46:33 UTC
(In reply to Jonathan Druart from comment #19)
> Created attachment 28614 [details] [review] [review]
> Bug 12279: Diacritics in system preference editor broken

Patch provided on bug 11944 (actually it just reverts this one).