Bug 25305

Summary: Double UTF-8 encoding on translation files
Product: Koha Reporter: Bernardo Gonzalez Kriegel <bgkriegel>
Component: I18N/L10NAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: 1joynelson, didier.gautheron, f.demians, jonathan.druart, martin.renvoize, victor
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25501
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00
Bug Depends on: 24217, 24365    
Bug Blocks: 25501    
Attachments: Bug 25305: fix double encoding in translation files
Bug 25305: fix double encoding in translation files
[ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding
Bug 25305: Translation process - Open all files specifying the utf8 encoding
Bug 25305: Translation process - Open all files specifying the utf8 encoding
Bug 25305: Translation process - Open all files specifying the utf8 encoding

Description Bernardo Gonzalez Kriegel 2020-04-28 17:57:05 UTC
Some translation files are showing double UTF-8 encoding, ej.

BokmÃ¥l instead of Bokmål
Bibliothèque intead of Bibliothèque,

and so on
Comment 1 Bernardo Gonzalez Kriegel 2020-04-28 18:09:44 UTC
Created attachment 103875 [details] [review]
Bug 25305: fix double encoding in translation files

Changes inroduced in Bug 24217 and Bug 24365 are
producing strings with double encoding

Don't know exactly what is producing the problem,
but I propose a fix

There are two changes, a feature limited use of
Modern::Perl and preventing double encoding in
output file.

To test:
1) Create translation files for a new language
( cd misc/translator; ./translate create xx-YY)
2) Verify double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check strange strings
3) Apply the patch
4) Create po files again
( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY)
5) Verify no more double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check normal strings
Comment 2 Victor Grousset/tuxayo 2020-04-29 06:27:16 UTC
Created attachment 103902 [details] [review]
Bug 25305: fix double encoding in translation files

Changes inroduced in Bug 24217 and Bug 24365 are
producing strings with double encoding

Don't know exactly what is producing the problem,
but I propose a fix

There are two changes, a feature limited use of
Modern::Perl and preventing double encoding in
output file.

To test:
1) Create translation files for a new language
( cd misc/translator; ./translate create xx-YY)
A new language means one that isn't already in Koha, xx-YY=>something you
invent.
2) Verify double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check strange strings
3) Apply the patch
4) Create po files again
( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY)
5) Verify no more double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check normal strings

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 3 Victor Grousset/tuxayo 2020-04-29 06:28:16 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #0)
> Some translation files are showing double UTF-8 encoding, ej.
> 
> BokmÃ¥l instead of Bokmål
> Bibliothèque intead of Bibliothèque,
> 
> and so on

Today I learned that this specific form of encoding issues has a name! Thanks Bernardo :) That might help when facing it in another context.


It works :D

I added the following note in the test plan:

> A new language means one that isn't already in Koha, xx-YY=>something you
> invent.
Comment 4 Bernardo Gonzalez Kriegel 2020-04-29 09:34:17 UTC
Thanks Victor :)
Comment 5 Jonathan Druart 2020-04-29 09:56:36 UTC
I am pretty sure this is not the correct fix, we are continuing to hide the true error.
Comment 6 Jonathan Druart 2020-04-29 09:57:25 UTC
Created attachment 103906 [details] [review]
[ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files specifying the utf8 encoding
Comment 7 Jonathan Druart 2020-04-29 09:57:47 UTC
(In reply to Jonathan Druart from comment #6)
> Created attachment 103906 [details] [review] [review]
> [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files
> specifying the utf8 encoding

What about this patch Bernardo?
Comment 8 Jonathan Druart 2020-04-29 09:59:53 UTC
Note: I missed one occurrence (not used in the normal workflow as it impacts stdout)

368 if (defined $output && $output ne '-') {
369     print STDERR "$0: Opening output file \"$output\"\n" if $verbose_p;
370     open($OUTPUT, '>:encoding(utf-8)', $output) || die "$output: $!\n";
371 } else {
372     print STDERR "$0: Outputting to STDOUT...\n" if $verbose_p;
373     open($OUTPUT, ">&STDOUT");
374 }
Comment 9 Bernardo Gonzalez Kriegel 2020-04-29 10:40:02 UTC
(In reply to Jonathan Druart from comment #5)
> I am pretty sure this is not the correct fix, we are continuing to hide the
> true error.

True! 

(In reply to Jonathan Druart from comment #7)
> (In reply to Jonathan Druart from comment #6)
> > Created attachment 103906 [details] [review] [review] [review]
> > [ALTERNATIVE-PATCH] Bug 25305: Translation process - Open all files
> > specifying the utf8 encoding
> 
> What about this patch Bernardo?

Way better, I'm only reverting, you are doing something :)
I'll test & sign
Comment 10 Bernardo Gonzalez Kriegel 2020-04-29 10:41:24 UTC
Created attachment 103914 [details] [review]
Bug 25305: Translation process - Open all files specifying the utf8 encoding

To test:
1) Create translation files for a new language
( cd misc/translator; ./translate create xx-YY)
A new language means one that isn't already in Koha, xx-YY=>something you
invent.
2) Verify double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check strange strings
3) Apply the patch
4) Create po files again
( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY)
5) Verify no more double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check normal string

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Copyied my test plan, can't find more cases, tested using
something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html
Comment 11 Jonathan Druart 2020-04-29 10:57:39 UTC
Great, thanks Bernardo!
Comment 12 Didier Gautheron 2020-05-02 10:28:19 UTC
Created attachment 104185 [details] [review]
Bug 25305: Translation process - Open all files specifying the utf8 encoding

To test:
1) Create translation files for a new language
( cd misc/translator; ./translate create xx-YY)
A new language means one that isn't already in Koha, xx-YY=>something you
invent.
2) Verify double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check strange strings
3) Apply the patch
4) Create po files again
( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY)
5) Verify no more double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check normal string

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Copyied my test plan, can't find more cases, tested using
something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html

Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com>
Comment 13 Didier Gautheron 2020-05-02 10:34:07 UTC
Also tested with a bogus UTF8 file:

iconv --verbose -f UTF8 -t ISO_8859-15 koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl > e.tt

mv e.tt koha-tmpl/intranet-tmpl/prog/en/xslt/NORMARCslim2intranetResults.xsl

( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY; echo $?)

egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
misc/translator/po/xx-YY-marc-MARC21.po:msgid "Norwegian (Bokmål)"
misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Andre typer periodika"
misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i b\\xF8ker eller periodika)"
misc/translator/po/xx-YY-marc-NORMARC.po:msgid "Artikler (i bøker eller periodika)"

Of course output is not proper UTF8 but there's no missing lines.
Comment 14 Katrin Fischer 2020-05-03 20:19:50 UTC
Created attachment 104226 [details] [review]
Bug 25305: Translation process - Open all files specifying the utf8 encoding

To test:
1) Create translation files for a new language
( cd misc/translator; ./translate create xx-YY)
A new language means one that isn't already in Koha, xx-YY=>something you
invent.
2) Verify double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check strange strings
3) Apply the patch
4) Create po files again
( cd misc/translator; rm -f po/xx-YY*; ./translate create xx-YY)
5) Verify no more double encoding
egrep "Aix-Marseille|Jean Prunier|periodika|Bokm" misc/translator/po/xx-YY-*
check normal string

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Copyied my test plan, can't find more cases, tested using
something similar to http://blogs.perl.org/users/chansen/2010/10/coping-with-double-encoded-utf-8.html

Signed-off-by: Didier Gautheron <didier.gautheron@biblibre.com>
Comment 15 Martin Renvoize 2020-05-04 07:55:32 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 16 Joy Nelson 2020-05-11 16:49:22 UTC
missing dependencies - not backported