Bug 28291

Summary: koha-translate install script producing incorrectly encoded YAML translation files
Product: Koha Reporter: Joonas Kylmälä <joonas.kylmala>
Component: Command-line UtilitiesAssignee: Joonas Kylmälä <joonas.kylmala>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: fridolin.somers, jonathan.druart, robin
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.05.00
Bug Depends on:    
Bug Blocks: 27623    
Attachments: Bug 28291: Make koha-translate script produce valid UTF-8 YAML files
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

Description Joonas Kylmälä 2021-05-05 13:12:19 UTC
The "koha-translate install" script AKA "misc/translator/translate install" script produces incorrectly encoded YAML files.

To reproduce:
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml

notice the file contains garbled characters.
Comment 1 Joonas Kylmälä 2021-05-05 14:28:05 UTC
Created attachment 120532 [details] [review]
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

We had two problems in the LangInstaller.pm module:
 1) the PO file was read as bytes instead of as a UTF-8 text stream
 2) The YAML file being outputted was double encoded, once by setting
    the file handle to output UTF-8 and other time in the DumpFile
    function internally

To test:
1. Before applying patch do the following
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml
2. Notice the output of the authorities_normal_marc21.yml contains
invalid looking UTF-8 characters
3. $ git clean -d -f # to remove the old translation files
4. Apply patch and repeat the steps and notice the
authorities_normal_marc21.yml contains valid looking UTF-8 characters
Comment 2 Joonas Kylmälä 2021-05-05 14:41:47 UTC
Moving this to needs sign-off given there is a manual test plan and the LangInstaller module modified here is only used by misc/translator/translate perl script...
Comment 3 Jonathan Druart 2021-05-06 13:00:38 UTC
Created attachment 120624 [details] [review]
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

We had two problems in the LangInstaller.pm module:
 1) the PO file was read as bytes instead of as a UTF-8 text stream
 2) The YAML file being outputted was double encoded, once by setting
    the file handle to output UTF-8 and other time in the DumpFile
    function internally

To test:
1. Before applying patch do the following
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml
2. Notice the output of the authorities_normal_marc21.yml contains
invalid looking UTF-8 characters
3. $ git clean -d -f # to remove the old translation files
4. Apply patch and repeat the steps and notice the
authorities_normal_marc21.yml contains valid looking UTF-8 characters

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 4 Kyle M Hall 2021-05-07 11:44:48 UTC
Created attachment 120685 [details] [review]
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

We had two problems in the LangInstaller.pm module:
 1) the PO file was read as bytes instead of as a UTF-8 text stream
 2) The YAML file being outputted was double encoded, once by setting
    the file handle to output UTF-8 and other time in the DumpFile
    function internally

To test:
1. Before applying patch do the following
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml
2. Notice the output of the authorities_normal_marc21.yml contains
invalid looking UTF-8 characters
3. $ git clean -d -f # to remove the old translation files
4. Apply patch and repeat the steps and notice the
authorities_normal_marc21.yml contains valid looking UTF-8 characters

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 5 Joonas Kylmälä 2021-05-07 11:48:48 UTC
Kyle, the attached patches don't contain your SO line, the same with bug 28281.
Comment 6 Jonathan Druart 2021-05-07 12:41:56 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 7 Fridolin Somers 2021-05-12 14:09:07 UTC
This depends on Bug 22824 no ?