|
Description
Bernardo Gonzalez Kriegel
2020-04-28 17:57:05 UTC
Created attachment 103875 [details] [review] Bug 25305: fix double encoding in translation files Changes inroduced in Bug 24217 and Bug 24365 are producing strings with double encoding Don't know exactly what is producing the problem, but I propose a fix There are two changes, a feature limited use of Modern::Perl and preventing double encoding in output file. To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal strings Created attachment 103902 [details] [review] Bug 25305: fix double encoding in translation files Changes inroduced in Bug 24217 and Bug 24365 are producing strings with double encoding Don't know exactly what is producing the problem, but I propose a fix There are two changes, a feature limited use of Modern::Perl and preventing double encoding in output file. To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal strings Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> (In reply to Bernardo Gonzalez Kriegel from comment #0) > Some translation files are showing double UTF-8 encoding, ej. > > BokmÃ¥l instead of Bokmål > Bibliothèque intead of Bibliothèque, > > and so on Today I learned that this specific form of encoding issues has a name! Thanks Bernardo :) That might help when facing it in another context. It works :D I added the following note in the test plan: > A new language means one that isn't already in Koha, xx-YY=>something you > invent. Thanks Victor :) I am pretty sure this is not the correct fix, we are continuing to hide the true error. Created attachment 103906 [details] [review] [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding (In reply to Jonathan Druart from comment #6) > Created attachment 103906 [details] [review] [review] > [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files > specifying the utf8 encoding What about this patch Bernardo? Note: I missed one occurrence (not used in the normal workflow as it impacts stdout)
368 if (defined $output && $output ne '-') {
369 print STDERR "$0: Opening output file \"$output\"\n" if $verbose_p;
370 open($OUTPUT, '>:encoding(utf-8)', $output) || die "$output: $!\n";
371 } else {
372 print STDERR "$0: Outputting to STDOUT...\n" if $verbose_p;
373 open($OUTPUT, ">&STDOUT");
374 }
(In reply to Jonathan Druart from comment #5) > I am pretty sure this is not the correct fix, we are continuing to hide the > true error. True! (In reply to Jonathan Druart from comment #7) > (In reply to Jonathan Druart from comment #6) > > Created attachment 103906 [details] [review] [review] [review] > > [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files > > specifying the utf8 encoding > > What about this patch Bernardo? Way better, I'm only reverting, you are doing something :) I'll test & sign Created attachment 103914 [details] [review] Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html Great, thanks Bernardo! Created attachment 104185 [details] [review] Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com> Also tested with a bogus UTF8 file: iconv --verbose -f UTF8 -t ISO_8859-15 koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl > e.tt mv e.tt koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY; echo $?) egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* misc/translator/po/xx-YY-marc-MARC21.po:msgid "Norwegian (Bokmål)" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Andre typer periodika" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i b\\xF8ker eller periodika)" misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i bøker eller periodika)" Of course output is not proper UTF8 but there's no missing lines. Created attachment 104226 [details] [review] Bug 25305: Translation process - Open all files specifying the utf8 encoding To test: 1) Create translation files for a new language ( cd misc/translator; ./translate create xx-YY) A new language means one that isn't already in Koha, xx-YY=>something you invent. 2) Verify double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check strange strings 3) Apply the patch 4) Create po files again ( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY) 5) Verify no more double encoding egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-* check normal string Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Copyied my test plan, can't find more cases, tested using something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com> Nice work everyone! Pushed to master for 20.05 missing dependencies - not backported |