Bug 10492

Summary: Translation problems with TT directives in po files
Product: Koha Reporter: Katrin Fischer <katrin.fischer>
Component: I18N/L10NAssignee: Bernardo Gonzalez Kriegel <bgkriegel>
Status: CLOSED FIXED QA Contact:
Severity: minor    
Priority: P5 - low CC: bgkriegel, f.demians, fridolin.somers, veron
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16559
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=16871
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.11.00,19.05.03
Attachments: Bug 10492: Translation problems with TT directives in po files
Bug 10492: Translation problems with TT directives in po files
Bug 10492: Translation problems with TT directives in po files

Description Katrin Fischer 2013-06-19 12:30:39 UTC
We have a few strings similar to that now appearing in the po files:

[%% UNLESS hidden.defined('streetnumber') && hidden.defined('address') && hidden.defined('address2') && hidden.defined('city') && hidden.defined('state') && hidden.defined('zipcode') && hidden.defined('country') %%] 

Those can't be translated correctly. If you change the [%% to [% the po file will be unusable and the translation scripts will give you an error. We need to fix the underlying template problem here.

http://translate.koha-community.org/de/312/translate.html#search=hidden.defined&sfields=source,target&unit=6514624
Comment 1 Katrin Fischer 2013-07-01 20:49:45 UTC
I have done some more tests on this and leaving the strings untouched will get you valid po files that also translate correctly in the templates. 

'[%%' will be correctly translated to '[%'.

But I am still wondering why the translation scripts pick up a lot of those strings now. Also there is a difference - some TT code has the escaped % some does not.

I am downgrading the bug, because if you are careful in your translations, you can create correct po files by just copying the strings without making any changes.
Comment 2 Katrin Fischer 2015-06-07 01:17:13 UTC
Taking a look at the 3.20 staff file, there are still a lot of [% showing up.
Comment 3 Katrin Fischer 2019-05-04 22:06:45 UTC
Searching for [% in the po files for 19.05 finds 69 results. A lot of those are like:

Koha [% Version %]

I am not sure why those exist yet. I would have expected Koha %s.
Comment 4 Bernardo Gonzalez Kriegel 2019-05-05 18:10:54 UTC
(In reply to Katrin Fischer from comment #3)
> Searching for [% in the po files for 19.05 finds 69 results. A lot of those
> are like:
> 
> Koha [% Version %]
> 
> I am not sure why those exist yet. I would have expected Koha %s.

They exist because they belong to 'attributes' in HTML tags, for example 'alt' in 'img', 'value' in inputs, or 'content' in 'meta'.
In this particular case, on line 5 of opac-tmpl/bootstrap/en/includes/doc-head-close.inc there is
<meta name="generator" content="Koha [% Version | html %]" />

Each line on each TT file is 'tokenized', and TTvariables are replaced with '%s'.
But an HTML tag (i.e. <meta ../>) is 'tokenized' as such, so no replacement is made there; only the text of some attributes is extracted in order to translate them.
Ideally we should no use TTvariables inside tag attributes, but it's useful :)

I'll try to write a patch to (partially) 'fix' this
Comment 5 Bernardo Gonzalez Kriegel 2019-05-05 19:14:00 UTC Comment hidden (obsolete)
Comment 6 Bernardo Gonzalez Kriegel 2019-05-05 19:19:24 UTC
Substitute
  koha-tmpl/intranet-tmpl/prog/es-ES
for
  koha-tmpl/intranet-tmpl/prog/xx-YY
on item 9) of test plan.
Comment 7 Owen Leonard 2019-07-31 17:09:35 UTC
Created attachment 91899 [details] [review]
Bug 10492: Translation problems with TT directives in po files

This patch implements the replacement of TTvariables
'[%...%]' with normal placeholders '%s' for text
inside HTML tags (eg. img, meta, input)

The replacement is done in two files:
* xgettext.pl, used on create/update, so TTvars are not
found on the text to translate (msgid)
* tmpl_proccess3.pl, used on install time, so the correct
translation is found (msgstr) and the right TTvar is put on
the translated text.

To test:
Before applying the patch
1) Update your favorite language
   cd misc/traslator
   perl translate update xx-YY

2) Build a list (ini.txt) of 'msgid' to do a comparison
   msgcat --no-wrap --use-first po/xx-YY-*po -o - | egrep "^msgid" | sort > ini.txt

3) Apply the patch

4) Update your favorite language, again
   cd misc/traslator
   perl translate update xx-YY

5) Build a final list (end.txt) of 'msgid' to do a comparison
   msgcat --no-wrap --use-first po/xx-YY-*po -o - | egrep "^msgid" | sort > end.txt

6) Compare both files, you must find (most) TTvars replaced by '%s'
   diff ini.txt end.txt

   for example:
   < msgid "Translate item type [% itemtype.itemtype | html %]"
   > msgid "Translate item type %s"

   < msgid "Holds on this item: [% item_loo.item_holds | html %] / Total holds on this record: [% item_loo.holds | html -%]"
   > msgid "Holds on this item: %s / Total holds on this record: %s"

   < msgid "tag_anchor_[% innerloo.tag | html %]_[% innerloo.index | html %][% innerloo.random | html %]"
   > msgid "tag_anchor_%s_%s%s"

   < msgid "[% innerloo.tag_lib | html %] - Click to Expand this Tag"
   > msgid "%s - Click to Expand this Tag"

7) Translate one of the examples, with more than one variable if
possible, for example this text in xx-YY-staff-prog.po
   > msgid "Holds on this item: %s / Total holds on this record: %s"
Check it's not marked as 'fuzzy'

8) Apply the translation
   perl translate install xx-YY

9) Check the translated string has all variables in the right order
   edit koha-tmpl/intranet-tmpl/prog/es-ES/modules/tools/batchMod-edit.tt
   around line '187', first '[% item_loo.item_holds | html %]', then
   '[% item_loo.holds | html -%]' on this case,
   or use another example

If approved this patch can be backported to current stable
versions.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 8 Nick Clemens 2019-08-02 13:55:58 UTC
Created attachment 91931 [details] [review]
Bug 10492: Translation problems with TT directives in po files

This patch implements the replacement of TTvariables
'[%...%]' with normal placeholders '%s' for text
inside HTML tags (eg. img, meta, input)

The replacement is done in two files:
* xgettext.pl, used on create/update, so TTvars are not
found on the text to translate (msgid)
* tmpl_proccess3.pl, used on install time, so the correct
translation is found (msgstr) and the right TTvar is put on
the translated text.

To test:
Before applying the patch
1) Update your favorite language
   cd misc/traslator
   perl translate update xx-YY

2) Build a list (ini.txt) of 'msgid' to do a comparison
   msgcat --no-wrap --use-first po/xx-YY-*po -o - | egrep "^msgid" | sort > ini.txt

3) Apply the patch

4) Update your favorite language, again
   cd misc/traslator
   perl translate update xx-YY

5) Build a final list (end.txt) of 'msgid' to do a comparison
   msgcat --no-wrap --use-first po/xx-YY-*po -o - | egrep "^msgid" | sort > end.txt

6) Compare both files, you must find (most) TTvars replaced by '%s'
   diff ini.txt end.txt

   for example:
   < msgid "Translate item type [% itemtype.itemtype | html %]"
   > msgid "Translate item type %s"

   < msgid "Holds on this item: [% item_loo.item_holds | html %] / Total holds on this record: [% item_loo.holds | html -%]"
   > msgid "Holds on this item: %s / Total holds on this record: %s"

   < msgid "tag_anchor_[% innerloo.tag | html %]_[% innerloo.index | html %][% innerloo.random | html %]"
   > msgid "tag_anchor_%s_%s%s"

   < msgid "[% innerloo.tag_lib | html %] - Click to Expand this Tag"
   > msgid "%s - Click to Expand this Tag"

7) Translate one of the examples, with more than one variable if
possible, for example this text in xx-YY-staff-prog.po
   > msgid "Holds on this item: %s / Total holds on this record: %s"
Check it's not marked as 'fuzzy'

8) Apply the translation
   perl translate install xx-YY

9) Check the translated string has all variables in the right order
   edit koha-tmpl/intranet-tmpl/prog/es-ES/modules/tools/batchMod-edit.tt
   around line '187', first '[% item_loo.item_holds | html %]', then
   '[% item_loo.holds | html -%]' on this case,
   or use another example

If approved this patch can be backported to current stable
versions.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 9 Martin Renvoize 2019-08-05 10:40:10 UTC
Nice work!

Pushed to master for 19.11.00
Comment 10 Fridolin Somers 2019-08-09 13:04:15 UTC
Pushed to 19.05.x for 19.05.03
Comment 11 Bernardo Gonzalez Kriegel 2019-11-19 16:55:11 UTC
*** Bug 19498 has been marked as a duplicate of this bug. ***