Created attachment 28316 [details] Screenshot of German OPAC tab in sys pref editor I see some strange behaviour with diacritics in the system preference editor on current master. What I did: - cd misc/translator - perl translate update de-DE - perl translate install de-DE The strange thing is that only some pages/umlauts are broken, while others on other tabs appear ok. I see the same behaviour with French - attaching screenshots.
Created attachment 28317 [details] Screenshot of French OPAC tab in sys pref editor
Created attachment 28318 [details] Screenshot of German Logs tab in sys pref editor And while OPAC is broken, the Logs tab displays diacritics correctly.
On a 3.15.00.052 it looks I have the same problem. I made cd misc/translator ./translate update de-DE ./translate install de-DE ./translate update fr-FR ./translate install fr-FR ./translate update tr-TR ./translate install tr-TR Some are broken, like opaccolorstylesheet : fr-FR: Inclure la feuille de style CSS supplémentaire pour outrepasser les paramètres spécifiques depuis la feuille de style par défaut (laisser vide pour désactiver). Inscrire juste le nom du fichier, un chemin local complet ou une URL complète commençant par http:// (si le fichier est sur un serveur distant). Merci de noter que si seul un nom de fichier est entré, le fichier devra se trouver dans le sous-dossier css pour chaque thème et langage actif dans le dossier des modèles de Koha. Un chemin local est attendu pour démarrer depuis votre fichier HTTP source. de-DE: Verwende zusätzlich das CSS-Stylesheet um Anweisungen im vorhandenen Stylesheet zu überschreiben (frei lassen, wenn nicht verwendet). Geben Sie nur einen Dateinamen oder einen vollständigen lokalen Pfad oder eine vollständige URL beginnend mit http:// an (falls die Datei auf einem anderen Server liegt). Wenn nur ein Dateiname angegeben wird, sollte die Datei im Verzeichnis css für das aktive Theme und die jeweilige Sprache gespeichert sein. Ein vollständiger lokaler Pfad beginnt mit dem Wurzelverzeichnis des Webservers. Some are OK, like language : fr-FR: Activer à l'interface professionnelle les langues suivantes : Deutsch(de-DE) English(en) Türkçe(tr-TR) Français(fr-FR) de-DE: Aktiviere folgende Sprachen in der Dienstoberfläche: Deutsch(de-DE) Français(fr-FR) Türkçe(tr-TR) English(en)
I reproduce with perl v5.18. Bernardo and Fridolin don't reproduce with perl v5.14.
Please report version of YAML::Syck Have no problems on Ubuntu 12.04 (ver 1.19), or Debian 7 (ver 1.20), but with latest version (ver 1.27, from cpan) I can reproduce
Yaml:Syck: 1.27 - It looks like we have a candidate.
Created attachment 28375 [details] [review] Bug 12279: Diacritics in system preference editor broken This patch changes how .pref files are read. Instead of going through all our global ultra-tweaked template object, we only use gettemplate to build the correct file name (path included) and load it using the (at functional) YAML::Syck::LoadFile method. Pros: Doesn't break anything for YAML::Syck pre-1.27 (the ones shipped with Ubuntu 14.04) Cons: something is wrong with C4::Templates encoding/decoding, or at least our default setup using C4::Templates->new doesn't work properly for all file types. To test: - On clean master, install a translation like: cd misc/translator ; perl translate install fr-FR - Go to sysprefs, opac => FAIL: descriptions are encoding-broken - Apply the patch, reload => SUCCESS: Everything shows ok Regards To+ Sponsored-by: Universidad Nacional de Cordoba
Created attachment 28377 [details] [review] Bug 12279: Diacritics in system preference editor broken This patch changes how .pref files are read. Instead of going through all our global ultra-tweaked template object, we only use gettemplate to build the correct file name (path included) and load it using the (at functional) YAML::Syck::LoadFile method. Pros: Doesn't break anything for YAML::Syck pre-1.27 (the ones shipped with Ubuntu 14.04) Cons: something is wrong with C4::Templates encoding/decoding, or at least our default setup using C4::Templates->new doesn't work properly for all file types. To test: - On clean master, install a translation like: cd misc/translator ; perl translate install fr-FR - Go to sysprefs, opac => FAIL: descriptions are encoding-broken - Apply the patch, reload => SUCCESS: Everything shows ok Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works ok, no errors Test: 1) with YAML::Syck 1.19, no problems before, translated preferences encode diacritics, ej. '\xC3\xA8' for grave accented e Page shows correctly 2) with YAML:Syck 1.27, translated preferences have unencoded diacritics, without patch display is broken, with patch it shows correctly
Created attachment 28378 [details] [review] [SIGNED-OFF] Bug 12279: Diacritics in system preference editor broken This patch changes how .pref files are read. Instead of going through all our global ultra-tweaked template object, we only use gettemplate to build the correct file name (path included) and load it using the (at functional) YAML::Syck::LoadFile method. Pros: Doesn't break anything for YAML::Syck pre-1.27 (the ones shipped with Ubuntu 14.04) Cons: something is wrong with C4::Templates encoding/decoding, or at least our default setup using C4::Templates->new doesn't work properly for all file types. To test: - On clean master, install a translation like: cd misc/translator ; perl translate install fr-FR - Go to sysprefs, opac => FAIL: descriptions are encoding-broken - Apply the patch, reload => SUCCESS: Everything shows ok Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works ok, no errors Test: 1) with YAML::Syck 1.19, no problems before, translated preferences encode diacritics, ej. '\xC3\xA8' for grave accented e Page shows correctly 2) with YAML:Syck 1.27, translated preferences have unencoded diacritics, without patch display is broken, with patch it shows correctly
Created attachment 28380 [details] [review] Bug 12279: Diacritics in system preference editor broken This patch changes how .pref files are read. Instead of going through all our global ultra-tweaked template object, we only use gettemplate to build the correct file name (path included) and load it using the (at functional) YAML::Syck::LoadFile method. Pros: Doesn't break anything for YAML::Syck pre-1.27 (the ones shipped with Ubuntu 14.04) Cons: something is wrong with C4::Templates encoding/decoding, or at least our default setup using C4::Templates->new doesn't work properly for all file types. To test: - On clean master, install a translation like: cd misc/translator ; perl translate install fr-FR - Go to sysprefs, opac => FAIL: descriptions are encoding-broken - Apply the patch, reload => SUCCESS: Everything shows ok Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works ok, no errors Test: 1) with YAML::Syck 1.19, no problems before, translated preferences encode diacritics, ej. '\xC3\xA8' for grave accented e Page shows correctly 2) with YAML:Syck 1.27, translated preferences have unencoded diacritics, without patch display is broken, with patch it shows correctly Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
YAML::Syck is v1.27 here and I confirm this patch fixes the issue. But I'm still wondering why the pref files contain a mix of '\xC3\xA8' and 'é'. Marked as Passed QA.
Two comments: This causes a regression, as a few of the tabs actually use TT: circulation.pref: - '[% local_currency %] in fines.' circulation.pref: - '[% local_currency %] in fines.' opac.pref: - '[% local_currency %] in fines (leave blank to disable).' patrons.pref: - '[% local_currency %].' Failing QA on that basis. Also, I'm not sure YAML::Syck 1.27 is specifically to blame -- on a Debian Wheezy system, when I installed YAML::Syck 1.27 from CPAN, I did not see the bug.
Created attachment 28424 [details] [review] Bug 12279: Diacritics in system preference editor broken This patch changes how .pref files are read. Instead of going through all our global ultra-tweaked C4::Templates->output method, it adds a new one that takes care of decoding the .pref file data for storing in memory. This might need to be re-done in the context of 11944, but at least provides a clean workaround for 3.16. To test: - On clean master, install a translation like: cd misc/translator ; perl translate install fr-FR - Go to sysprefs, opac => FAIL: descriptions are encoding-broken - Apply the patch, reload => SUCCESS: Everything shows ok Regards To+ Sponsored-by: Universidad Nacional de Cordoba
Created attachment 28438 [details] [review] Bug 12279: Diacritics in system preference editor broken This patch changes how .pref files are read. Instead of going through all our global ultra-tweaked C4::Templates->output method, it adds a new one that takes care of decoding the .pref file data for storing in memory. This might need to be re-done in the context of 11944, but at least provides a clean workaround for 3.16. To test: - On clean master, install a translation like: cd misc/translator ; perl translate install fr-FR - Go to sysprefs, opac => FAIL: descriptions are encoding-broken - Apply the patch, reload => SUCCESS: Everything shows ok Regards To+ Sponsored-by: Universidad Nacional de Cordoba Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 28517 [details] [review] Bug 12279: Counterpatch touching only preferences.pl Instead of a new routine in Templates.pm that actually only decodes the data, I would recommend to put this temporary fix into preferences.pl. Bug 11944 should provide the real fix..
Moving status to Failed QA in order to make a decision between both approaches.
(In reply to M. de Rooy from comment #15) > Created attachment 28517 [details] [review] [review] > Bug 12279: Counterpatch touching only preferences.pl > > Instead of a new routine in Templates.pm that actually only decodes the > data, I would recommend to put this temporary fix into preferences.pl. > Bug 11944 should provide the real fix.. I first tried that same approach, but hesitated to implement it that way beacuse i wasn't sure about the effect of double encoding in some unknown scenario: i.e. we are messing with UTF-8 inside C4::Templates->output() and again on its output. Anyway, I tried this (again) on both YAML::Syck 1.19 and 1.27 and the patch fixes the problem. If Galen agrees on Marcel's approach I'd like to push this one soon.
Created attachment 28557 [details] [review] Bug 12279: Diacritics in system preference editor broken Temporary fix into preferences.pl. Bug 11944 should provide the real fix.. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 28614 [details] [review] Bug 12279: Diacritics in system preference editor broken Temporary fix into preferences.pl. Bug 11944 should provide the real fix.. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Patch pushed to the master branch. Thanks Marcel!
Pushed to 3.16.x.
(In reply to Jonathan Druart from comment #19) > Created attachment 28614 [details] [review] [review] > Bug 12279: Diacritics in system preference editor broken Patch provided on bug 11944 (actually it just reverts this one).