This report replaces bug 6554.
*** Bug 6554 has been marked as a duplicate of this bug. ***
Created attachment 23154 [details] [review]
Bug 11302: Template::output should deal with objects
if a ref is not HASH or ARRAY, C4::Template::output assumes it is a
scalar. Which is wrong, it could be an object.
At least the member 'description' of the C4::Category object should be
Before applying the patch
- translate a template for a language
- create a patron category with a no-ASCII character
- got the patron home page and select the translated language
- you should see a broken encoding
Apply the patch
- refresh the page
- all looks good now
When I apply this patch, I get failure on the plugins page:
Not a HASH reference at /home/boutros/Projects/deichman/kohaclone/C4/Templates.pm line 158.
Note that I have installed a plugin with non-ascii characters, so that might be why
First observation: If I put a diacritical char in the branchname, the staff home page will show a mangled char in the upper-right corner (directly after login). If you refresh after login or come back there, it will disappear.
Some (!) additional observations:
Facets in staff search show mangled chars.
Patron category (with latin diacritic and chinese char) on memberentry.pl mangled.
Branch name mangled too on memberentry.pl
Catalogue detail: mangled chars all over the place.
I was just a try. If it does not work, I reopen the initial report and close this one.
Created attachment 26151 [details] [review]
Bug 11302: Template::output should deal with objects
If a ref is not HASH or ARRAY, C4::Template::output assumes it
is a scalar. Which is wrong, it could be an object.
Note: this is yet another / alternative take on the problem,
with only bare minimum changes made to the code base, so
this (counter-)patch - while not pretty - hopefully should be
less regression-prone. Right now, the only white-listed object
this patch deals with is 'Koha::AdditionalField' introduced
by bug 10855. In case it tests OK, without any regressions, we can
add another objects (C4::Category etc.) later on.
1) apply bug 10855 first; observe that non-ascii characters
(in authorized category names or additional field labels) are not
2) apply patch,
3) ensure that encoding of non-ascii characters for features added
by 10855 got fixed,
4) there should be no character-encoding-related regressions
of any kind appearing anywhere in the system.
Sorry Jacek your simpler approach doesn't work for authorized values with UTF-8 characters in them (in fact, it doesn't even return double encoded utf-8, it just doesn't return anything which is somewhat puzzling since I can't figure out from code why it does so).
(In reply to Dobrica Pavlinusic from comment #8)
> Sorry Jacek your simpler approach doesn't work for authorized values with
> UTF-8 characters in them (in fact, it doesn't even return double encoded
> utf-8, it just doesn't return anything which is somewhat puzzling since I
> can't figure out from code why it does so).
It took me a while to reproduce this issue, but indeed there seems to be at least one serious problem with this particular approach, which is affecting bug 10855 in non intuitive way (I'm not entirely sure it's the same problem you've encountered - it can be yet another/new one as well). Anyway, it boils down to this:
- unpatched C4/Template.pm does not encode hash keys, and does not deal with objects at all
- patched C4/Template.pm does deal with whitelisted objects, but it only encodes scalar values within them (in the same exact manner as unpatched version deals with ordinary, unblessed hashes)
- if there are utf8 characters in additional field names, some code fragments introduced by 10855 - e.g. this one, used repeatedly in subscription-add.tt
wouldn't work any longer. While field.name (being a scalar value) gets encoded by patched C4/Template.pm, keys in additional_fields hash do not.
What's worse, I don't see any easy solution for this problem (other than making some - pretty substantial - changes to 10855). We can patch Template.pm somehow further, so it will encode hash keys - but that's rather risky move IMO; who knows how many regressions we may get elsewhere in the system..
I suspect that touching encoding in C4::Template::output is wrong direction to take.
I would rather decode utf-8 as soon as it arrives in Koha and have correct encoding within perl code and then let TT deal with encoding it back to binary bytes by using binmode( STDOUT, ":utf8" ); instead of traversing all hashes we are passing to TT which has performance impact.
I will extend http://wiki.koha-community.org/wiki/Handling_UTF8_in_development to include more information about it and prepare initial patch with replaces usage of utf8::(en|de)code witn Encode::(en|de)code so we can see if this approach is feasible.
(In reply to Dobrica Pavlinusic from comment #10)
> I will extend
> http://wiki.koha-community.org/wiki/Handling_UTF8_in_development to include
> more information about it and prepare initial patch with replaces usage of
> utf8::(en|de)code witn Encode::(en|de)code so we can see if this approach is
It would be really great to get rid of all those numerous ::[en|de]code, html_entity and so on.. Sounds like a pretty ambitious task, though. If I understand correctly, essentially it would be something like bug 6554 revival (which was expected for 3.14 but unfortunately got reverted), only better this time. Hopefully it will finally be included in 3.16 ;)
This patch does not fix correctly the issue. See bug 11944 for the next step.