Bug 28291 - koha-translate install script producing incorrectly encoded YAML translation files
Summary: koha-translate install script producing incorrectly encoded YAML translation ...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Joonas Kylmälä
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 27623
  Show dependency treegraph
 
Reported: 2021-05-05 13:12 UTC by Joonas Kylmälä
Modified: 2021-12-13 21:12 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.05.00


Attachments
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files (2.27 KB, patch)
2021-05-05 14:28 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files (2.34 KB, patch)
2021-05-06 13:00 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files (2.36 KB, patch)
2021-05-07 11:44 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Joonas Kylmälä 2021-05-05 13:12:19 UTC
The "koha-translate install" script AKA "misc/translator/translate install" script produces incorrectly encoded YAML files.

To reproduce:
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml

notice the file contains garbled characters.
Comment 1 Joonas Kylmälä 2021-05-05 14:28:05 UTC
Created attachment 120532 [details] [review]
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

We had two problems in the LangInstaller.pm module:
 1) the PO file was read as bytes instead of as a UTF-8 text stream
 2) The YAML file being outputted was double encoded, once by setting
    the file handle to output UTF-8 and other time in the DumpFile
    function internally

To test:
1. Before applying patch do the following
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml
2. Notice the output of the authorities_normal_marc21.yml contains
invalid looking UTF-8 characters
3. $ git clean -d -f # to remove the old translation files
4. Apply patch and repeat the steps and notice the
authorities_normal_marc21.yml contains valid looking UTF-8 characters
Comment 2 Joonas Kylmälä 2021-05-05 14:41:47 UTC
Moving this to needs sign-off given there is a manual test plan and the LangInstaller module modified here is only used by misc/translator/translate perl script...
Comment 3 Jonathan Druart 2021-05-06 13:00:38 UTC
Created attachment 120624 [details] [review]
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

We had two problems in the LangInstaller.pm module:
 1) the PO file was read as bytes instead of as a UTF-8 text stream
 2) The YAML file being outputted was double encoded, once by setting
    the file handle to output UTF-8 and other time in the DumpFile
    function internally

To test:
1. Before applying patch do the following
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml
2. Notice the output of the authorities_normal_marc21.yml contains
invalid looking UTF-8 characters
3. $ git clean -d -f # to remove the old translation files
4. Apply patch and repeat the steps and notice the
authorities_normal_marc21.yml contains valid looking UTF-8 characters

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 4 Kyle M Hall 2021-05-07 11:44:48 UTC
Created attachment 120685 [details] [review]
Bug 28291: Make koha-translate script produce valid UTF-8 YAML files

We had two problems in the LangInstaller.pm module:
 1) the PO file was read as bytes instead of as a UTF-8 text stream
 2) The YAML file being outputted was double encoded, once by setting
    the file handle to output UTF-8 and other time in the DumpFile
    function internally

To test:
1. Before applying patch do the following
$ cd misc/translator
$ ./translate update pl-PL
$ ./translate install pl-PL
$ cd -
$ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml
2. Notice the output of the authorities_normal_marc21.yml contains
invalid looking UTF-8 characters
3. $ git clean -d -f # to remove the old translation files
4. Apply patch and repeat the steps and notice the
authorities_normal_marc21.yml contains valid looking UTF-8 characters

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 5 Joonas Kylmälä 2021-05-07 11:48:48 UTC
Kyle, the attached patches don't contain your SO line, the same with bug 28281.
Comment 6 Jonathan Druart 2021-05-07 12:41:56 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 7 Fridolin Somers 2021-05-12 14:09:07 UTC
This depends on Bug 22824 no ?