Bug 11290

Summary: members-home.pl displays badly if using non english templates and have a utf8 char in categorycode description
Product: Koha Reporter: Chris Cormack <chris>
Component: I18N/L10NAssignee: Bugs List <koha-bugs>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: abl, f.demians, fridolin.somers, gitbot, gmcharlt, jonathan.druart, katrin.fischer, kyle, m.de.rooy, tomascohen
Version: master   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=6554
http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=11302
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 11290 : Fixing display problem on members-home.pl
Bug 11290 : Fixing display problem on members-home.pl
[PASSED QA] Bug 11290 : Fixing display problem on members-home.pl

Description Chris Cormack 2013-11-22 22:08:03 UTC

    
Comment 1 Chris Cormack 2013-11-22 22:09:45 UTC
What are the steps to reproduce
1) In a fresh Koha, generate translation into XX
2) Select XX for the staff interface (i.e. enable it, and choose it)
3) Go to the Patrons page
4) Everything looks fine
5) In the admin page, create a patron category with (say) the non-ASCII character 'ñ'.
6) Go back to the Patrons page, everything is broken
Comment 2 Chris Cormack 2013-11-22 22:12:48 UTC Comment hidden (obsolete)
Comment 3 I'm just a bot 2013-11-22 22:44:24 UTC
Patch applied cleanly, go forth and signoff
Comment 4 Tomás Cohen Arazi 2013-11-23 02:28:18 UTC Comment hidden (obsolete)
Comment 5 Katrin Fischer 2013-11-24 21:36:17 UTC
Created attachment 23122 [details] [review]
[PASSED QA] Bug 11290 : Fixing display problem on members-home.pl

To test
1) In a fresh Koha, generate translation into XX
2) Select XX for the staff interface (i.e. enable it, and choose it)
3) Go to the Patrons page
4) Everything looks fine
5) In the admin page, create a patron category with any no-ASCII character.
6) Go back to the Patrons page, everything is broken
7) apply patch
8) regenerate translation
9) go back to the Patrons page, everything looks fine
10) switch to english it still looks fine

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Tried with ru-RU and fixes the problem.

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes all tests and QA script.
Problem is also visible on untranslated templates changing
one of the sample category descriptions to include diacritics.
Comment 6 Marcel de Rooy 2013-11-25 08:44:50 UTC
Just an additional observation:

This seems to work for me too. I tested a few combinations including Latin diacritics, and some Chinese characters in the patron category as well as in the members template text. The following remark on Apache::Util did however not fully reassure me:

Apache::Util, escape_html says: [This function will correctly escape US-ASCII output. If you are using a different character set such as UTF8, or need more control on the escaping process, use HTML::Entities.]

Note that Template Toolkit (at least version 2.87) does however first try Apache::Util and only if that is not available, HTML::Entities.

Just under the assumption that HTML::Entities would do a better job, we could do the following, forcing TT to use it:
use Template::Filters;
Template::Filters->use_html_entities;
(That would make HTML::Entities a Koha dependency.)
Comment 7 Jacek Ablewicz 2013-11-25 11:02:07 UTC
Thank you for the patch, it's working for me as well.

That said, I wonder if instead of adding more and more html_entity to various templates (there is an similar problem with, e.g., bug 10855, and probably more like that), shouldn't we rather fix C4/Templates.pm?
I think the underlying cause for this particular problem in members.pl and members-home.pl is that output()/utf8_arrayref()/utf8_hashref() subs in Templates.pm do not know what to do with objects / blessed hashrefs (they treat them as strings/scalars, actually calling utf8::is_utf8() on them etc.).
Comment 8 Marcel de Rooy 2013-11-25 12:06:33 UTC
(In reply to Jacek Ablewicz from comment #7)
> That said, I wonder if instead of adding more and more html_entity to
> various templates (there is an similar problem with, e.g., bug 10855, and
> probably more like that), shouldn't we rather fix C4/Templates.pm?
> I think the underlying cause for this particular problem in members.pl and
> members-home.pl is that output()/utf8_arrayref()/utf8_hashref() subs in
> Templates.pm do not know what to do with objects / blessed hashrefs (they
> treat them as strings/scalars, actually calling utf8::is_utf8() on them
> etc.).

+1 for a generic solution (if available :)
Comment 9 Jacek Ablewicz 2013-11-25 16:56:06 UTC
As an experiment I replaced ref() calls in C4/Templates.pm with reftype() from Scalar::Util package; objects/blessed hashrefs are now processed by output(). But it has an unexepected side effect - in every place html_entity filter is being used (we have 3 such places in koha-tmpl/.../intranet/ ) encoding of filtered part now seems to be broken (removing html_entity filter from templates, or replacing it with html filter fixes it). Or perhaps I'm missing something here, and such side efect is acutally being expected (?).

BTW, html_entity filter is also used in modules/pay.tt - and encoding of this part ([% line.title |html_entity %] ) indeed seems to be broken (same symptoms in unpatched 3.12.4 version too, not just in master/3.15 with patched or unpatched Templates.pm). Does anyone remeber when and way was is added there?
Comment 10 Katrin Fischer 2013-11-25 21:08:39 UTC
I am also in favor of a generic solution that would help avoid these problems in the future, but I think by pushing this we will get rid of a prominently showing problem quickly. It would give us time to work on the big solution, that will probably require much more time for testing.
Comment 11 Tomás Cohen Arazi 2013-11-26 12:08:14 UTC
(In reply to Jacek Ablewicz from comment #7)
> I think the underlying cause for this particular problem in members.pl and
> members-home.pl is that output()/utf8_arrayref()/utf8_hashref() subs in
> Templates.pm do not know what to do with objects / blessed hashrefs (they
> treat them as strings/scalars, actually calling utf8::is_utf8() on them
> etc.).

Preparing for Reno i had tried even removing any call to those sub. I'd be in favour of removing them and using template filters instead (I.e. less loops traversing data structures and less probable double encoding situations)..

regards
to+
Comment 12 Chris Cormack 2013-11-26 18:33:27 UTC
(In reply to Tomás Cohen Arazi from comment #11)
> (In reply to Jacek Ablewicz from comment #7)
> > I think the underlying cause for this particular problem in members.pl and
> > members-home.pl is that output()/utf8_arrayref()/utf8_hashref() subs in
> > Templates.pm do not know what to do with objects / blessed hashrefs (they
> > treat them as strings/scalars, actually calling utf8::is_utf8() on them
> > etc.).
> 
> Preparing for Reno i had tried even removing any call to those sub. I'd be
> in favour of removing them and using template filters instead (I.e. less
> loops traversing data structures and less probable double encoding
> situations)..
> 
I would prefer this situation also
Comment 13 Galen Charlton 2013-12-10 01:25:03 UTC
As a stop-gap, I've pushed this to master.  Thanks, Chris!
Comment 14 Fridolin Somers 2014-05-07 16:31:38 UTC
Pushed to 3.14.x, will be in 3.14.07
Comment 15 Kyle M Hall 2014-06-06 13:48:53 UTC
Pushed to 3.12.x will be in 3.12.13