Bug 25501

Summary: Encoding issues in the translation process
Product: Koha Reporter: Bernardo Gonzalez Kriegel <bgkriegel>
Component: I18N/L10NAssignee: Julian Maurice <julian.maurice>
Status: CLOSED FIXED QA Contact: Jonathan Druart <jonathan.druart>
Severity: critical    
Priority: P5 - low CC: 1joynelson, f.demians, jonathan.druart, julian.maurice, katrin.fischer, martin.renvoize, victor
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25305
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00
Bug Depends on: 25305    
Bug Blocks:    
Attachments: Bug 25501: use utf8 for open in tmpl_process3.pl
Bug 25501: use utf8 for open in tmpl_process3.pl
Bug 25501: Encode strings in utf8 for output - translation process
Bug 25501: Pass the encoding to Locale::PO
Bug 25501: Encode strings in utf8 for output - translation process
Bug 25501: Pass the encoding to Locale::PO
Bug 25501: Encode strings in utf8 for output - translation process
Bug 25501: Pass the encoding to Locale::PO
Bug 25501: Remove local vars
Bug 25501: Remove 'Wide character in print' for po2json
Bug 25501: Supress warnings on installing translation
Bug 25501: Supress warnings on installing translation

Description Bernardo Gonzalez Kriegel 2020-05-14 11:32:14 UTC
Install any translation, 

(cd misc/translator/; ./translate install fr-FR )
 see example output

Wide character in print at /../misc/translator/tmpl_process3.pl line 144.
Wide character in printf at /.../misc/translator/tmpl_process3.pl line 141.
Wide character in print at /.../misc/translator/tmpl_process3.pl line 134
Comment 1 Bernardo Gonzalez Kriegel 2020-05-14 11:38:33 UTC
Created attachment 104876 [details] [review]
Bug 25501: use utf8 for open in tmpl_process3.pl

Opening files using "":encoding(UTF-8)" removes
the warning "wide character in print" installing
translations

To test:
1) Verify the error installing translations for any language
2) Apply the patch
3) Repeat 1), check warning is gone
Comment 2 Jonathan Druart 2020-05-14 12:18:20 UTC
I am used to explicitly specify the encoding when opening the file, but I am fine with this approach as well.
Comment 3 Victor Grousset/tuxayo 2020-05-18 18:33:07 UTC
Created attachment 105043 [details] [review]
Bug 25501: use utf8 for open in tmpl_process3.pl

Opening files using "":encoding(UTF-8)" removes
the warning "wide character in print" installing
translations

To test:
1) Verify the error installing translations for any language
2) Apply the patch
3) Repeat 1), check warning is gone

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 4 Jonathan Druart 2020-05-19 07:11:11 UTC
Bernardo, there is an encoding issue in the translation process now.
Translated interface in French: Périodiques
Comment 5 Jonathan Druart 2020-05-19 08:12:06 UTC
Created attachment 105060 [details] [review]
Bug 25501: Encode strings in utf8 for output - translation process

This is a follow-up of bug 25305, all strings need to be utf8 encoded
when printed to the file.

Test plan:
Test the whole translation process and translate a language you know.
Test the translated interface for: XSLT (detail and result pages), syspref, and "normal" views
You should not get encoding errors (or warning in logs), nowhere.
Comment 6 Jonathan Druart 2020-05-19 10:12:06 UTC
Created attachment 105067 [details] [review]
Bug 25501: Pass the encoding to Locale::PO
Comment 7 Jonathan Druart 2020-05-19 10:12:46 UTC
Still not ready:

misc/translator/po/fr-FR-installer-MARC21.po: GNU gettext message catalogue, ASCII text
misc/translator/po/fr-FR-installer.po:        GNU gettext message catalogue, ASCII text
misc/translator/po/fr-FR-marc-MARC21.po:      GNU gettext message catalogue, UTF-8 Unicode text
misc/translator/po/fr-FR-marc-NORMARC.po:     GNU gettext message catalogue, UTF-8 Unicode text
misc/translator/po/fr-FR-marc-UNIMARC.po:     GNU gettext message catalogue, UTF-8 Unicode text
misc/translator/po/fr-FR-messages-js.po:      GNU gettext message catalogue, UTF-8 Unicode text
misc/translator/po/fr-FR-messages.po:         GNU gettext message catalogue, UTF-8 Unicode text
misc/translator/po/fr-FR-opac-bootstrap.po:   GNU gettext message catalogue, UTF-8 Unicode text
misc/translator/po/fr-FR-pref.po:             GNU gettext message catalogue, UTF-8 Unicode text, with very long lines
misc/translator/po/fr-FR-staff-prog.po:       GNU gettext message catalogue, UTF-8 Unicode text, with very long lines
Comment 8 Jonathan Druart 2020-05-19 10:17:39 UTC
Actually better than before, for message-js:

misc/translator/po/fr-FR-messages-js.po:      GNU gettext message catalogue, ASCII text


the installer PO files are not translated yet, and so does not contain utf8 chars.

Ready for testing.
Comment 9 Jonathan Druart 2020-05-19 10:31:29 UTC
Created attachment 105073 [details] [review]
Bug 25501: Encode strings in utf8 for output - translation process

This is a follow-up of bug 25305, all strings need to be utf8 encoded
when printed to the file.

Test plan:
Test the whole translation process and translate a language you know.
Test the translated interface for: XSLT (detail and result pages), syspref, and "normal" views
You should not get encoding errors (or warning in logs), nowhere.
Comment 10 Jonathan Druart 2020-05-19 10:31:34 UTC
Created attachment 105074 [details] [review]
Bug 25501: Pass the encoding to Locale::PO
Comment 11 Bernardo Gonzalez Kriegel 2020-05-19 11:00:08 UTC
With my patch there were 624 files with double encoding after install :(
With yours none

One qa error, you need to remove line 380 in misc/translator/tmpl_process3.pl
-        local(*INPUT, *OUTPUT);

* But now something new: *

(cd misc/translator/; perl translate update es-ES; perl translate install es-ES)

Wide character in print at /home/bgkriegel/koha/kohaclone/misc/translator/po2json line 171.

this never ends...
Comment 12 Jonathan Druart 2020-05-19 11:10:12 UTC
Created attachment 105080 [details] [review]
Bug 25501: Encode strings in utf8 for output - translation process

This is a follow-up of bug 25305, all strings need to be utf8 encoded
when printed to the file.

Test plan:
Test the whole translation process and translate a language you know.
Test the translated interface for: XSLT (detail and result pages), syspref, and "normal" views
You should not get encoding errors (or warning in logs), nowhere.
Comment 13 Jonathan Druart 2020-05-19 11:10:17 UTC
Created attachment 105081 [details] [review]
Bug 25501: Pass the encoding to Locale::PO
Comment 14 Jonathan Druart 2020-05-19 11:12:12 UTC
Created attachment 105084 [details] [review]
Bug 25501: Remove local vars
Comment 15 Jonathan Druart 2020-05-19 11:15:10 UTC
Created attachment 105085 [details] [review]
Bug 25501: Remove 'Wide character in print' for po2json
Comment 16 Julian Maurice 2020-05-19 11:29:47 UTC
Here is a simpler counter patch : https://gitlab.com/jajm/Koha/-/commit/a8764d0b0add70acbf5d5b5ad6046c1571e2ec13

It suppresses all warnings for me and I haven't found any encoding issues in the interface yet
Comment 17 Bernardo Gonzalez Kriegel 2020-05-19 13:13:20 UTC
Jonathan,

Without patch, no double encoding but warnings
With patches, no warnings but 635 files with double encoding
With Julian's approach, no warnings nor double encoding

Checked encoding issues using http://paste.koha-community.org/20552
Comment 18 Bernardo Gonzalez Kriegel 2020-05-19 13:40:41 UTC
Created attachment 105095 [details] [review]
Bug 25501: Supress warnings on installing translation

To test:
1) Verify the error installing translations for any language
2) Apply the patch
3) Repeat 1), check warning is gone

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Attributed to Julian
No warnings, no double encoding, no errors.
Comment 19 Jonathan Druart 2020-05-19 14:10:39 UTC
Created attachment 105099 [details] [review]
Bug 25501: Supress warnings on installing translation

To test:
1) Verify the error installing translations for any language
2) Apply the patch
3) Repeat 1), check warning is gone

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Attributed to Julian
No warnings, no double encoding, no errors.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Tested full install with yml and js strings, everything works great!
Comment 20 Jonathan Druart 2020-05-19 14:11:41 UTC
Lowering severity as it did not actually break things.
Comment 21 Martin Renvoize 2020-05-19 14:21:37 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 22 Julian Maurice 2020-05-19 14:34:04 UTC
(In reply to Jonathan Druart from comment #20)
> Lowering severity as it did not actually break things.

I'm not sure if you mean that encoding issues are minor issues, or that there wasn't any encoding issue in the first place, but the patch did fix a double encoding issue for me.
Comment 23 Jonathan Druart 2020-05-19 15:35:58 UTC
(In reply to Julian Maurice from comment #22)
> (In reply to Jonathan Druart from comment #20)
> > Lowering severity as it did not actually break things.
> 
> I'm not sure if you mean that encoding issues are minor issues, or that
> there wasn't any encoding issue in the first place, but the patch did fix a
> double encoding issue for me.

On master?
I got encoding issues with Bernardo's attempt, but did not notice something wrong on master, did you?
Comment 24 Katrin Fischer 2020-05-19 16:13:37 UTC
I had noticed problems with XSLT not displaying and the wide character error for de-DE on master a few days ago. Can retest now with the patch.
Comment 25 Julian Maurice 2020-05-20 06:25:59 UTC
(In reply to Jonathan Druart from comment #23)
> On master?
> I got encoding issues with Bernardo's attempt, but did not notice something
> wrong on master, did you?

Yes. To reproduce:
1. git reset --hard 8994ded2c47d22fd46ea46f6e7855da52826b351 # the commit before 25501
2. cd misc/translator && ./translate install fr-FR
3. Open koha-tmpl/intranet-tmpl/prog/fr-FR/xslt/MARC21Languages.xsl, you should see the string "Hébreu"
4. git reset --hard c97921bb68849a2c7dd5f2f5ad8f5b97789df5c1 
5. ./translate install fr-FR
6. Reopen koha-tmpl/intranet-tmpl/prog/fr-FR/xslt/MARC21Languages.xsl and see that the string is now "Hébreu"
Comment 26 Katrin Fischer 2020-05-20 06:39:42 UTC
(In reply to Katrin Fischer from comment #24)
> I had noticed problems with XSLT not displaying and the wide character error
> for de-DE on master a few days ago. Can retest now with the patch.

I just retested and things see to be ok now - thx for all the work here!
Comment 27 Joy Nelson 2020-05-20 21:08:14 UTC
not backported to 19.11.x due to missing dependencies