Bug 28154

Summary: Translate script faces encoding issues
Product: Koha Reporter: Jonathan Druart <jonathan.druart>
Component: I18N/L10NAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: f.demians, fridolin.somers, joonas.kylmala, julian.maurice, martin.renvoize
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=28281
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.05.00
Circulation function:
Bug Depends on: 22824, 25067    
Bug Blocks: 27623    
Attachments: Bug 28154: Fix encoding issues on Koha-pref.pot
Bug 28154: Fix encoding issues on Koha-pref.pot
Bug 28154: Fix encoding issues on Koha-installer.pot
Bug 28154: Fix encoding issues on Koha-pref.pot
Bug 28154: Fix encoding issues on Koha-installer.pot

Description Jonathan Druart 2021-04-15 14:57:57 UTC
% gulp po:update --lang pl-PL
misc/translator/Koha-pref.pot:5085:37: invalid multibyte sequence
misc/translator/Koha-pref.pot:5245:107: invalid multibyte sequence
misc/translator/Koha-pref.pot:5253:78: invalid multibyte sequence
misc/translator/Koha-pref.pot:5261:82: invalid multibyte sequence
msgmerge: found 4 fatal errors
Comment 1 Jonathan Druart 2021-04-15 14:58:54 UTC
Git bisect blames 
  commit d6d01169509999e22fbf33d839a41e402cb93a38
  Bug 22824: Replace YAML::Syck with YAML::XS
Comment 2 Jonathan Druart 2021-04-15 15:00:11 UTC
% file misc/translator/Koha-pref.pot
misc/translator/Koha-pref.pot: GNU gettext message catalogue, Non-ISO extended-ASCII text, with very long lines
Comment 3 Jonathan Druart 2021-04-15 15:01:57 UTC
Created attachment 119636 [details] [review]
Bug 28154: Fix encoding issues on Koha-pref.pot
Comment 4 Jonathan Druart 2021-04-15 15:03:15 UTC
Maybe this occurrence needs to be fixed as well:
misc/translator/xgettext-installer:Locale::PO->save_file_fromhash($output, $pot);
Comment 5 Jonathan Druart 2021-04-15 15:03:57 UTC
Cannot focus more on this one for today.
Comment 6 Julian Maurice 2021-04-16 07:42:35 UTC
Created attachment 119695 [details] [review]
Bug 28154: Fix encoding issues on Koha-pref.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 7 Julian Maurice 2021-04-16 07:42:40 UTC
Created attachment 119696 [details] [review]
Bug 28154: Fix encoding issues on Koha-installer.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 8 Julian Maurice 2021-04-16 07:43:12 UTC
(In reply to Jonathan Druart from comment #4)
> Maybe this occurrence needs to be fixed as well:
> misc/translator/xgettext-installer:Locale::PO->save_file_fromhash($output,
> $pot);

Yep, you were right. Fixed in the 2nd patch
Comment 9 Martin Renvoize (ashimema) 2021-04-16 11:47:58 UTC
Created attachment 119709 [details] [review]
Bug 28154: Fix encoding issues on Koha-pref.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize (ashimema) 2021-04-16 11:48:02 UTC
Created attachment 119710 [details] [review]
Bug 28154: Fix encoding issues on Koha-installer.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Martin Renvoize (ashimema) 2021-04-16 11:48:17 UTC
Good catch, Passing QA
Comment 12 Jonathan Druart 2021-04-16 11:56:24 UTC
(In reply to Julian Maurice from comment #8)
> (In reply to Jonathan Druart from comment #4)
> > Maybe this occurrence needs to be fixed as well:
> > misc/translator/xgettext-installer:Locale::PO->save_file_fromhash($output,
> > $pot);
> 
> Yep, you were right. Fixed in the 2nd patch

Can you detail what it fixes?
Comment 13 Julian Maurice 2021-04-16 12:56:37 UTC
(In reply to Jonathan Druart from comment #12)
> Can you detail what it fixes?

There is no real problem right now as there is no non-ASCII character in installer files, but if you add one you will see encoding issues.
For instance, add a 'é' to installer/data/mysql/en/mandatory/auth_values.yml 

  - "Default Koha system authorised values é"

and run `gulp po:update --lang pl-PL

You should see

misc/translator/Koha-installer.pot:972:46: Séquence d'octets multiples invalide
msgmerge: 1 erreur fatale trouvée                       
[14:55:39] Finished 'po_update_installer' after 152 ms

After the patch, there should be no errors
Comment 14 Jonathan Druart 2021-04-16 13:07:23 UTC
Right, thanks!
I've tried yesterday with an other file and not in the description. Now I recreate it.
Comment 15 Jonathan Druart 2021-04-16 13:12:23 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 16 Fridolin Somers 2021-04-22 14:20:57 UTC
Can this be pushed to 20.11.x even it does not contain Bug 22824 ?