The "koha-translate install" script AKA "misc/translator/translate install" script produces incorrectly encoded YAML files. To reproduce: $ cd misc/translator $ ./translate update pl-PL $ ./translate install pl-PL $ cd - $ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml notice the file contains garbled characters.
Created attachment 120532 [details] [review] Bug 28291: Make koha-translate script produce valid UTF-8 YAML files We had two problems in the LangInstaller.pm module: 1) the PO file was read as bytes instead of as a UTF-8 text stream 2) The YAML file being outputted was double encoded, once by setting the file handle to output UTF-8 and other time in the DumpFile function internally To test: 1. Before applying patch do the following $ cd misc/translator $ ./translate update pl-PL $ ./translate install pl-PL $ cd - $ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml 2. Notice the output of the authorities_normal_marc21.yml contains invalid looking UTF-8 characters 3. $ git clean -d -f # to remove the old translation files 4. Apply patch and repeat the steps and notice the authorities_normal_marc21.yml contains valid looking UTF-8 characters
Moving this to needs sign-off given there is a manual test plan and the LangInstaller module modified here is only used by misc/translator/translate perl script...
Created attachment 120624 [details] [review] Bug 28291: Make koha-translate script produce valid UTF-8 YAML files We had two problems in the LangInstaller.pm module: 1) the PO file was read as bytes instead of as a UTF-8 text stream 2) The YAML file being outputted was double encoded, once by setting the file handle to output UTF-8 and other time in the DumpFile function internally To test: 1. Before applying patch do the following $ cd misc/translator $ ./translate update pl-PL $ ./translate install pl-PL $ cd - $ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml 2. Notice the output of the authorities_normal_marc21.yml contains invalid looking UTF-8 characters 3. $ git clean -d -f # to remove the old translation files 4. Apply patch and repeat the steps and notice the authorities_normal_marc21.yml contains valid looking UTF-8 characters Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Created attachment 120685 [details] [review] Bug 28291: Make koha-translate script produce valid UTF-8 YAML files We had two problems in the LangInstaller.pm module: 1) the PO file was read as bytes instead of as a UTF-8 text stream 2) The YAML file being outputted was double encoded, once by setting the file handle to output UTF-8 and other time in the DumpFile function internally To test: 1. Before applying patch do the following $ cd misc/translator $ ./translate update pl-PL $ ./translate install pl-PL $ cd - $ less installer/data/mysql/pl-PL/marcflavour/marc21/mandatory/authorities_normal_marc21.yml 2. Notice the output of the authorities_normal_marc21.yml contains invalid looking UTF-8 characters 3. $ git clean -d -f # to remove the old translation files 4. Apply patch and repeat the steps and notice the authorities_normal_marc21.yml contains valid looking UTF-8 characters Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Kyle, the attached patches don't contain your SO line, the same with bug 28281.
Pushed to master for 21.05, thanks to everybody involved!
This depends on Bug 22824 no ?