Bug 28154 - Translate script faces encoding issues
Summary: Translate script faces encoding issues
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL:
Keywords:
Depends on: 22824 25067
Blocks: 27623
  Show dependency treegraph
 
Reported: 2021-04-15 14:57 UTC by Jonathan Druart
Modified: 2021-05-04 09:35 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:
21.05.00


Attachments
Bug 28154: Fix encoding issues on Koha-pref.pot (743 bytes, patch)
2021-04-15 15:01 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 28154: Fix encoding issues on Koha-pref.pot (801 bytes, patch)
2021-04-16 07:42 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 28154: Fix encoding issues on Koha-installer.pot (759 bytes, patch)
2021-04-16 07:42 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 28154: Fix encoding issues on Koha-pref.pot (866 bytes, patch)
2021-04-16 11:47 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 28154: Fix encoding issues on Koha-installer.pot (824 bytes, patch)
2021-04-16 11:48 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2021-04-15 14:57:57 UTC
% gulp po:update --lang pl-PL
misc/translator/Koha-pref.pot:5085:37: invalid multibyte sequence
misc/translator/Koha-pref.pot:5245:107: invalid multibyte sequence
misc/translator/Koha-pref.pot:5253:78: invalid multibyte sequence
misc/translator/Koha-pref.pot:5261:82: invalid multibyte sequence
msgmerge: found 4 fatal errors
Comment 1 Jonathan Druart 2021-04-15 14:58:54 UTC
Git bisect blames 
  commit d6d01169509999e22fbf33d839a41e402cb93a38
  Bug 22824: Replace YAML::Syck with YAML::XS
Comment 2 Jonathan Druart 2021-04-15 15:00:11 UTC
% file misc/translator/Koha-pref.pot
misc/translator/Koha-pref.pot: GNU gettext message catalogue, Non-ISO extended-ASCII text, with very long lines
Comment 3 Jonathan Druart 2021-04-15 15:01:57 UTC
Created attachment 119636 [details] [review]
Bug 28154: Fix encoding issues on Koha-pref.pot
Comment 4 Jonathan Druart 2021-04-15 15:03:15 UTC
Maybe this occurrence needs to be fixed as well:
misc/translator/xgettext-installer:Locale::PO->save_file_fromhash($output, $pot);
Comment 5 Jonathan Druart 2021-04-15 15:03:57 UTC
Cannot focus more on this one for today.
Comment 6 Julian Maurice 2021-04-16 07:42:35 UTC
Created attachment 119695 [details] [review]
Bug 28154: Fix encoding issues on Koha-pref.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 7 Julian Maurice 2021-04-16 07:42:40 UTC
Created attachment 119696 [details] [review]
Bug 28154: Fix encoding issues on Koha-installer.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 8 Julian Maurice 2021-04-16 07:43:12 UTC
(In reply to Jonathan Druart from comment #4)
> Maybe this occurrence needs to be fixed as well:
> misc/translator/xgettext-installer:Locale::PO->save_file_fromhash($output,
> $pot);

Yep, you were right. Fixed in the 2nd patch
Comment 9 Martin Renvoize 2021-04-16 11:47:58 UTC
Created attachment 119709 [details] [review]
Bug 28154: Fix encoding issues on Koha-pref.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize 2021-04-16 11:48:02 UTC
Created attachment 119710 [details] [review]
Bug 28154: Fix encoding issues on Koha-installer.pot

Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Martin Renvoize 2021-04-16 11:48:17 UTC
Good catch, Passing QA
Comment 12 Jonathan Druart 2021-04-16 11:56:24 UTC
(In reply to Julian Maurice from comment #8)
> (In reply to Jonathan Druart from comment #4)
> > Maybe this occurrence needs to be fixed as well:
> > misc/translator/xgettext-installer:Locale::PO->save_file_fromhash($output,
> > $pot);
> 
> Yep, you were right. Fixed in the 2nd patch

Can you detail what it fixes?
Comment 13 Julian Maurice 2021-04-16 12:56:37 UTC
(In reply to Jonathan Druart from comment #12)
> Can you detail what it fixes?

There is no real problem right now as there is no non-ASCII character in installer files, but if you add one you will see encoding issues.
For instance, add a 'é' to installer/data/mysql/en/mandatory/auth_values.yml 

  - "Default Koha system authorised values é"

and run `gulp po:update --lang pl-PL

You should see

misc/translator/Koha-installer.pot:972:46: Séquence d'octets multiples invalide
msgmerge: 1 erreur fatale trouvée                       
[14:55:39] Finished 'po_update_installer' after 152 ms

After the patch, there should be no errors
Comment 14 Jonathan Druart 2021-04-16 13:07:23 UTC
Right, thanks!
I've tried yesterday with an other file and not in the description. Now I recreate it.
Comment 15 Jonathan Druart 2021-04-16 13:12:23 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 16 Fridolin Somers 2021-04-22 14:20:57 UTC
Can this be pushed to 20.11.x even it does not contain Bug 22824 ?