Bug 24365

Summary: Using strict from TmplTokenizer.pm broke the translator script
Product: Koha Reporter: Jonathan Druart <jonathan.druart>
Component: I18N/L10NAssignee: Jonathan Druart <jonathan.druart>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: 1joynelson, f.demians, martin.renvoize
Version: unspecified   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00
Circulation function:
Bug Depends on: 24217    
Bug Blocks: 25305    
Attachments: Bug 24365: (bug 24217 follow-up) Fix encoding issue for translator script

Description Jonathan Druart 2020-01-07 14:50:36 UTC 
I am not sure what happened here, but adding strict for TmplTokenizer.pm breaks a test for translation:

kohadev-koha@5dbfd5c26b19:/kohadevbox/koha$ prove xt/author/translatable-templates.t
xt/author/translatable-templates.t .. 1/? 
#   Failed test 'opac_bootstrap templates are translatable'
#   at xt/author/translatable-templates.t line 91.
# /tmp/nUR3OD9D9y:19466:8: invalid multibyte sequence
#
# msgmerge: found 1 fatal error
#
xt/author/translatable-templates.t .. 2/? 
#   Failed test 'staff_prog templates are translatable'
#   at xt/author/translatable-templates.t line 91.
# /tmp/DdbDRQTBzl:6111:13: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:6122:13: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:10911:14: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:11301:14: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:49459:66: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:65223:27: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:88587:10: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:94544:43: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:94544:71: invalid multibyte sequence
#
# /tmp/DdbDRQTBzl:94549:8: invalid multibyte sequence
#
# msgmerge: found 10 fatal errors
#
# Looks like you failed 2 tests of 2.
xt/author/translatable-templates.t .. Dubious, test returned 2 (wstat 512, 0x200)
Failed 2/2 subtests 

Test Summary Report
-------------------
xt/author/translatable-templates.t (Wstat: 512 Tests: 2 Failed: 2)
  Failed tests:  1-2
  Non-zero exit status: 2
Files=1, Tests=2,  8 wallclock secs ( 0.02 usr  0.01 sys +  8.55 cusr  0.08 csys =  8.66 CPU)
Result: FAIL
Comment 1 Jonathan Druart 2020-01-07 14:54:15 UTC 
Created attachment 96896 [details] [review]
Bug 24365: (bug 24217 follow-up) Fix encoding issue for translator script

I am not sure what happened here, but adding strict for TmplTokenizer.pm
breaks a test for translation.

It has been caught by a failing test:
 % prove xt/author/translatable-templates.t
will display lines "invalid multibyte sequence"

I did a diff between the generated files and found:
< msgid "Ã rbok"
---
> msgid "Årbok"

Which made me think of an encoding issue.
Specifying the encoding when write open the file fixes the problem.

Test plan:
 % prove xt/author/translatable-templates.t
must return green
Comment 2 Martin Renvoize (ashimema) 2020-01-07 15:07:02 UTC 
Trivial fix, PQA and Pushing
Comment 3 Martin Renvoize (ashimema) 2020-01-07 15:07:51 UTC 
Nice work everyone!

Pushed to master for 20.05
Comment 4 Joy Nelson 2020-01-14 01:05:06 UTC 
not backported to 19.11.x