Some translation files are showing double UTF-8 encoding, ej. BokmÃ¥l instead of Bokmål Bibliothèque intead of Bibliothèque, and so on
Created attachment 103875 [details] [review] Bug 25305: fix double encoding in translation files Changes inroduced in Bug 24217 and Bug 24365 are producing strings with double encoding Don't know exactly what is producing the problem, but I propose a fix There are two changes, a feature limited use of Modern::Perl and preventing double encoding in output file. To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal strings
Created attachment 103902 [details] [review] Bug 25305: fix double encoding in translation files Changes inroduced in Bug 24217 and Bug 24365 are producing strings with double encoding Don't know exactly what is producing the problem, but I propose a fix There are two changes, a feature limited use of Modern::Perl and preventing double encoding in output file. To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal strings Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
(In reply to Bernardo Gonzalez Kriegel from comment #0) > Some translation files are showing double UTF-8 encoding, ej. > > BokmÃ¥l instead of Bokmål > Bibliothèque intead of Bibliothèque, > > and so on Today I learned that this specific form of encoding issues has a name! Thanks Bernardo :) That might help when facing it in another context. It works :D I added the following note in the test plan: > A new language means one that isn't already in Koha, xx-YY=>something you > invent.
Thanks Victor :)
I am pretty sure this is not the correct fix, we are continuing to hide the true error.
Created attachment 103906 [details] [review] [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding
(In reply to Jonathan Druart from comment #6) > Created attachment 103906 [details] [review] [review] > [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files > specifying the utf8 encoding What about this patch Bernardo?
Note: I missed one occurrence (not used in the normal workflow as it impacts stdout) 368 if (defined $output && $output ne '-') { 369 print STDERR "$0: Opening output file \"$output\"\n" if $verbose_p; 370 open($OUTPUT, '>:encoding(utf-8)', $output) || die "$output: $!\n"; 371 } else { 372 print STDERR "$0: Outputting to STDOUT...\n" if $verbose_p; 373 open($OUTPUT, ">&STDOUT"); 374 }
(In reply to Jonathan Druart from comment #5) > I am pretty sure this is not the correct fix, we are continuing to hide the > true error. True! (In reply to Jonathan Druart from comment #7) > (In reply to Jonathan Druart from comment #6) > > Created attachment 103906 [details] [review] [review] [review] > > [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files > > specifying the utf8 encoding > > What about this patch Bernardo? Way better, I'm only reverting, you are doing something :) I'll test & sign
Created attachment 103914 [details] [review] Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html
Great, thanks Bernardo!
Created attachment 104185 [details] [review] Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com>
Also tested with a bogus UTF8 file: iconv --verbose -f UTF8 -t ISO_8859-15 koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl > e.tt mv e.tt koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY; echo $?) egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* misc/translator/po/xx-YY-marc-MARC21.po:msgid "Norwegian (Bokmål)" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Andre typer periodika" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i b\\xF8ker eller periodika)" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i bøker eller periodika)" Of course output is not proper UTF8 but there's no missing lines.
Created attachment 104226 [details] [review] Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com>
Nice work everyone! Pushed to master for 20.05
missing dependencies - not backported