Bug 11944 - Cleanup Koha UTF-8
Summary: Cleanup Koha UTF-8
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Jonathan Druart
QA Contact: Galen Charlton
URL:
Keywords:
: 10068 10075 (view as bug list)
Depends on: 11248 11906 13264
Blocks: 12832 13345 22070 7380 10855 11844 12176 12637 13239 13577 13579 13596 13646 13810 13815 13934 14431 14813 14817 15570 15818 17676
  Show dependency treegraph
 
Reported: 2014-03-14 15:31 UTC by Jonathan Druart
Modified: 2021-01-21 11:21 UTC (History)
17 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Large patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 11944: Remove all utf8 filter from templates (12.20 KB, patch)
2014-03-14 19:19 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.69 KB, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (728 bytes, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (709 bytes, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.59 KB, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.11 KB, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.61 KB, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.70 KB, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.19 KB, patch)
2014-03-14 19:20 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.33 KB, patch)
2014-03-17 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (1.93 KB, patch)
2014-03-17 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching - latin characters (1.97 KB, patch)
2014-03-17 15:15 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (12.20 KB, patch)
2014-04-02 07:25 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.32 KB, patch)
2014-04-02 07:25 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (728 bytes, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (709 bytes, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.59 KB, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.11 KB, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.11 KB, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.61 KB, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.70 KB, patch)
2014-04-02 07:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.19 KB, patch)
2014-04-02 07:27 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.05 KB, patch)
2014-04-02 07:27 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching - latin characters (1.97 KB, patch)
2014-04-02 07:27 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.88 KB, patch)
2014-04-03 14:24 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (12.21 KB, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.60 KB, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (728 bytes, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (709 bytes, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.59 KB, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.11 KB, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.61 KB, patch)
2014-04-07 15:11 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.70 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.19 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.05 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.88 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.58 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.30 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Facets and resort (3.78 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Authentication (2.01 KB, patch)
2014-04-07 15:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.64 KB, patch)
2014-04-08 10:24 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.02 KB, patch)
2014-04-09 13:25 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (2.43 KB, patch)
2014-04-11 13:47 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.30 KB, patch)
2014-04-14 09:57 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (2.97 KB, patch)
2014-04-14 14:37 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.76 KB, patch)
2014-04-16 08:30 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.12 KB, patch)
2014-04-16 08:30 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.64 KB, patch)
2014-04-16 09:15 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.28 KB, patch)
2014-04-16 12:47 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.72 KB, patch)
2014-04-16 12:48 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.12 KB, patch)
2014-04-16 12:48 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.64 KB, patch)
2014-04-16 12:48 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (12.21 KB, patch)
2014-04-16 14:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.60 KB, patch)
2014-04-16 14:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (728 bytes, patch)
2014-04-16 14:12 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (709 bytes, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.11 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.61 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.70 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.19 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.05 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.88 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.58 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.30 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Facets and resort (3.78 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Authentication (2.01 KB, patch)
2014-04-16 14:13 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.02 KB, patch)
2014-04-16 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.28 KB, patch)
2014-04-16 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.72 KB, patch)
2014-04-16 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.12 KB, patch)
2014-04-16 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.64 KB, patch)
2014-04-16 14:14 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (12.26 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.65 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (779 bytes, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (761 bytes, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.16 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.66 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.75 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.24 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.10 KB, patch)
2014-04-25 09:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.93 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.63 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.35 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Facets and resort (3.83 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Authentication (2.06 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.07 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.33 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (188.77 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.17 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.69 KB, patch)
2014-04-25 09:46 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (12.25 KB, patch)
2014-05-01 16:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.77 KB, patch)
2014-05-01 16:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (841 bytes, patch)
2014-05-01 16:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (823 bytes, patch)
2014-05-01 16:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.22 KB, patch)
2014-05-01 16:19 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.67 KB, patch)
2014-05-01 16:35 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.81 KB, patch)
2014-05-01 16:35 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.30 KB, patch)
2014-05-01 16:35 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.17 KB, patch)
2014-05-01 16:35 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.99 KB, patch)
2014-05-01 16:36 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.69 KB, patch)
2014-05-01 16:36 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.41 KB, patch)
2014-05-01 16:36 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Facets and resort (3.89 KB, patch)
2014-05-01 16:36 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Authentication (2.06 KB, patch)
2014-05-01 16:43 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.13 KB, patch)
2014-05-01 16:43 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.38 KB, patch)
2014-05-01 16:44 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (184.98 KB, patch)
2014-05-01 16:44 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.23 KB, patch)
2014-05-01 16:44 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (2.01 KB, patch)
2014-05-01 16:49 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (12.46 KB, patch)
2014-05-08 10:17 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.04 KB, patch)
2014-05-08 10:17 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (843 bytes, patch)
2014-05-08 10:17 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (825 bytes, patch)
2014-05-08 10:17 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.23 KB, patch)
2014-05-08 10:17 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.66 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.81 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.31 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.20 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.99 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.69 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.41 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Facets and resort (3.89 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Authentication (2.12 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.13 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.39 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (190.09 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.23 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.75 KB, patch)
2014-05-08 10:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on search history (1.63 KB, patch)
2014-05-08 10:50 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove bad FIXME in guided_report (1.66 KB, patch)
2014-05-08 11:02 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Facets and resort (1.53 KB, patch)
2014-05-08 14:07 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (190.02 KB, patch)
2014-05-14 08:35 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.38 KB, patch)
2014-05-14 08:35 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.75 KB, patch)
2014-05-14 08:36 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on search history (1.63 KB, patch)
2014-05-14 08:36 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove bad FIXME in guided_report (1.66 KB, patch)
2014-05-14 08:36 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Facets and resort (1.53 KB, patch)
2014-05-14 08:36 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue in C4::ItemType (1.77 KB, patch)
2014-05-14 09:26 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: (follow-up) Remove all utf8 filter from templates (2.61 KB, patch)
2014-05-14 09:32 UTC, Jonathan Druart
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 11944: (follow-up) Remove all utf8 filter from templates (2.85 KB, patch)
2014-05-14 09:40 UTC, Magnus Enger
Details | Diff | Splinter Review
Bug 11944: (follow-up) Remove all utf8 filter from templates (2.84 KB, patch)
2014-06-09 10:37 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (11.30 KB, patch)
2014-07-23 16:05 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (11.09 KB, patch)
2014-07-23 16:05 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (913 bytes, patch)
2014-07-23 16:07 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (895 bytes, patch)
2014-07-23 16:07 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.28 KB, patch)
2014-07-23 16:08 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.12 KB, patch)
2014-07-23 16:12 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.88 KB, patch)
2014-07-23 16:13 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.33 KB, patch)
2014-07-23 16:13 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.70 KB, patch)
2014-07-23 16:13 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (11.84 KB, patch)
2014-07-23 16:13 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.75 KB, patch)
2014-07-23 16:13 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.46 KB, patch)
2014-07-23 16:14 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Authentication (2.11 KB, patch)
2014-07-23 16:16 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.18 KB, patch)
2014-07-23 16:17 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.43 KB, patch)
2014-07-23 16:17 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (185.93 KB, patch)
2014-07-23 16:17 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.43 KB, patch)
2014-07-23 16:17 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.42 KB, patch)
2014-07-23 16:20 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on search history (1.62 KB, patch)
2014-07-23 16:22 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Remove bad FIXME in guided_report (1.71 KB, patch)
2014-07-23 16:22 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Facets and resort (751 bytes, patch)
2014-07-23 16:23 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue in C4::ItemType (1.84 KB, patch)
2014-07-23 16:23 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: (follow-up) Remove all utf8 filter from templates (2.88 KB, patch)
2014-07-23 16:24 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Revert "Bug 12279: Diacritics in system preference editor broken" (1.12 KB, patch)
2014-07-23 16:24 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: remove url/uri filter from query_cgi (7.21 KB, patch)
2014-07-23 16:25 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: decode uri_unescape data (2.02 KB, patch)
2014-07-23 16:25 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 11944: Convert DB tables to utf8_unicode_ci (1.15 KB, patch)
2014-07-24 08:27 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Convert DB tables to utf8_unicode_ci (67.26 KB, patch)
2014-07-24 09:10 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix suggestion at the OPAC for perl v5.10 (1.40 KB, patch)
2014-08-18 15:35 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci (2.49 KB, patch)
2014-08-29 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove all utf8 filter from templates (11.50 KB, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace use of utf8 with Encode (10.88 KB, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Encoding UTF8 in C4::Templates (993 bytes, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Set ImplicitUnicode for YAML (994 bytes, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Library name variable should be decoded (1.42 KB, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere (10.21 KB, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Never use decode/encode in TT Plugins (1.98 KB, patch)
2015-01-14 15:42 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove obsolete encoding routines (2.44 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: TT Plugins should not encode strings (2.81 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Some refresh don't indicate the charset to use (12.10 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: FIX Error on editing an authority (1.86 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on searching (2.58 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Authentication (2.22 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix fund names encoding on the parcel page (3.30 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on adding a subscription (5.56 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: use CGI( -utf8 ) everywhere (187.81 KB, patch)
2015-01-14 15:43 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding on sending emails (5.53 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue for the "Did you mean" feature (1.54 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue on search history (1.73 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Remove bad FIXME in guided_report (1.39 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix encoding issue in C4::ItemType (1.93 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: (follow-up) Remove all utf8 filter from templates (3.01 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Revert "Bug 12279: Diacritics in system preference editor broken" (1.23 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: remove url/uri filter from query_cgi (7.38 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: decode uri_unescape data (2.13 KB, patch)
2015-01-14 15:44 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Convert DB tables to utf8_unicode_ci (68.09 KB, patch)
2015-01-14 15:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix suggestion at the OPAC for perl v5.10 (1.53 KB, patch)
2015-01-14 15:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci (2.60 KB, patch)
2015-01-14 15:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: revert unneeded IsStringUTF8ish behaviour change (693 bytes, patch)
2015-01-14 15:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: Fix problems introduced by bug 11515 (4.07 KB, patch)
2015-01-14 15:45 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 11944: (RM followup) Missing dependency in C4::Context (570 bytes, patch)
2015-01-14 15:45 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Jonathan Druart 2014-03-14 15:31:22 UTC
This bug report will try to fix all UTF-8 problems once and forever!

see http://wiki.koha-community.org/wiki/Handling_UTF8_in_development for more details about development guidelines for handling UTF-8 in Koha.
Comment 1 Tomás Cohen Arazi 2014-03-14 15:49:21 UTC
What would you guys think of pushing

use feature 'unicode_strings'
or
use 5.012

and then fixing possible breakages?
Comment 2 Jonathan Druart 2014-03-14 19:19:24 UTC Comment hidden (obsolete)
Comment 3 Jonathan Druart 2014-03-14 19:20:03 UTC Comment hidden (obsolete)
Comment 4 Jonathan Druart 2014-03-14 19:20:08 UTC Comment hidden (obsolete)
Comment 5 Jonathan Druart 2014-03-14 19:20:12 UTC Comment hidden (obsolete)
Comment 6 Jonathan Druart 2014-03-14 19:20:20 UTC Comment hidden (obsolete)
Comment 7 Jonathan Druart 2014-03-14 19:20:24 UTC Comment hidden (obsolete)
Comment 8 Jonathan Druart 2014-03-14 19:20:28 UTC Comment hidden (obsolete)
Comment 9 Jonathan Druart 2014-03-14 19:20:31 UTC Comment hidden (obsolete)
Comment 10 Jonathan Druart 2014-03-14 19:20:35 UTC Comment hidden (obsolete)
Comment 11 Jonathan Druart 2014-03-14 19:23:20 UTC
(In reply to Tomás Cohen Arazi from comment #1)
> What would you guys think of pushing
> 
> use feature 'unicode_strings'
> or
> use 5.012
> 
> and then fixing possible breakages?

Hi Tomás, we seen your comment too late, sorry :-/
I will have a look at unicode_strings ASAP!
Comment 12 Jonathan Druart 2014-03-14 19:24:31 UTC
I think this could be tested in order to catch bugs and have feedback quickly.
Known bug: on importing an utf8 notice, the imported notice is broken.
Comment 13 Jacek Ablewicz 2014-03-15 10:48:22 UTC
URI de-/encoding needs some additional work - current version has serious uri encoding-related issues. To reproduce: try to search for some term (containing utf8 characters) in OPAC. Note: I tested OPAC searching with bootstrap theme (not sure if it may be relevant or not).

E.g., searching for 'życie':

/cgi-bin/koha/opac-search.pl?idx=&q=%C5%BCycie

does return expected results, but links pointing to further/refined searches (facets, sorting, ..) on the result page are broken (pagination links are fine, though - probably because they are not being encoded at all). Offending links do have extra %25 's in unicode byte sequence:

/cgi-bin/koha/opac-search.pl?idx=kw&q=%25C5%25BCycie&sort_by=relevance_asc&limit=itype:BK-KP
Comment 14 Jonathan Druart 2014-03-17 14:14:38 UTC Comment hidden (obsolete)
Comment 15 Jonathan Druart 2014-03-17 14:14:44 UTC Comment hidden (obsolete)
Comment 16 Jonathan Druart 2014-03-17 15:15:54 UTC Comment hidden (obsolete)
Comment 17 Zeno Tajoli 2014-03-31 14:22:19 UTC
I insert a list of test for this bug:

-- Install it on a marc21/english setup
-- Import from files and from z39.50 greek/cyrillic records
-- Index them in zebra, try searches and factes
-- Catalogue update/display of record with greek/cyrillic chars
-- Indexing update rsults
-- libraries with greek/cyrillic names
-- Try extra fields for patron with special chars [with system preference
ExtendedPatronAttribute on 'YES']
-- Greek/cyrillic chars in ACQ budget/ordee/vendor
-- Greek/cyrillic chars in letters
Comment 18 Paola Rossi 2014-04-01 15:16:24 UTC
I test against master 3.15.00.26

The second patch doesn't apply:

Applying: Bug 11944: replace use of utf8 with Encode
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging Koha/Database.pm
CONFLICT (content): Merge conflict in Koha/Database.pm
Failed to merge in the changes.

This is ./Koha/Database.pm:

    60      my $schema    = Koha::Schema->connect(
    61          "DBI:$db_driver:dbname=$db_name;host=$db_host;port=$db_port",
    62  <<<<<<< HEAD
    63          $db_user, $db_passwd, $db_opts );
    64  =======
    65          $db_user, $db_passwd );
    66
    67  >>>>>>> Bug 11944: replace use of utf8 with Encode
    68      return $schema;
    69  }
Comment 19 Jonathan Druart 2014-04-02 07:25:51 UTC Comment hidden (obsolete)
Comment 20 Jonathan Druart 2014-04-02 07:25:58 UTC Comment hidden (obsolete)
Comment 21 Jonathan Druart 2014-04-02 07:26:02 UTC Comment hidden (obsolete)
Comment 22 Jonathan Druart 2014-04-02 07:26:08 UTC Comment hidden (obsolete)
Comment 23 Jonathan Druart 2014-04-02 07:26:13 UTC Comment hidden (obsolete)
Comment 24 Jonathan Druart 2014-04-02 07:26:18 UTC Comment hidden (obsolete)
Comment 25 Jonathan Druart 2014-04-02 07:26:47 UTC Comment hidden (obsolete)
Comment 26 Jonathan Druart 2014-04-02 07:26:51 UTC Comment hidden (obsolete)
Comment 27 Jonathan Druart 2014-04-02 07:26:56 UTC Comment hidden (obsolete)
Comment 28 Jonathan Druart 2014-04-02 07:27:00 UTC Comment hidden (obsolete)
Comment 29 Jonathan Druart 2014-04-02 07:27:04 UTC Comment hidden (obsolete)
Comment 30 Jonathan Druart 2014-04-02 07:27:09 UTC Comment hidden (obsolete)
Comment 31 Paola Rossi 2014-04-02 15:27:43 UTC
I apply against 3.15.00.026.

In "Authorized values" of Administration module, I add a "New category", whose name is (newcat-mu):
newcat-μ

1) ERROR:
Soon after I save it, 3 greek chars are shown badly here:
"New authorized values for newcat-μ:"
"Authorized values for category newcat-μ:"
"There are no authorized values defined for newcat-μ"

The current URL is bad too:
http://192.168.40.128:8080/cgi-bin/koha/admin/authorised_values.pl?searchfield=newcat-μ&offset=0

2) OK:
The new category has been added to the list "Show category". Its name is shown OK.
Now I select the new category "newcat-μ" inside this list, and I select "Show category".
Koha answers with a page where all the 5 occurrences of newcat-μ are shown OK.

So I pass to Failed QA status.
Comment 32 Jonathan Druart 2014-04-02 15:45:47 UTC
(In reply to Paola Rossi from comment #31)
> In "Authorized values" of Administration module, I add a "New category",
> whose name is (newcat-mu):
> newcat-μ
> 
> 1) ERROR:
> Soon after I save it, 3 greek chars are shown badly here:
> "New authorized values for newcat-μ:"
> "Authorized values for category newcat-μ:"
> "There are no authorized values defined for newcat-μ"

Hum, weird, I did not reproduce on my local instance. Same on sandbox 17: 
http://pro.test17.biblibre.com/cgi-bin/koha/admin/authorised_values.pl

Templates fr-FR and it-IT generated.
Comment 33 Paola Rossi 2014-04-02 16:11:38 UTC
Dear Mr. Jonathan, 

two images are available at:

1)http://koha.wdfiles.com/local--files/immaginibugs/newcat-mu.png
2)http://koha.wdfiles.com/local--files/immaginibugs/newcat2-mu.png

The first image shows the error, while the second one shows correct chars.

The second one shows the already added category.
The first one shows the moment soon after the adding.

Your test at comment 32 references to the second moment, where no error is shown.

When I cancel the category too, I see wrong chars.

In the Sandbox 17, I recreated the error, see link:
3) http://www.koha.it/local--files/immaginibugs/newcat3-mu.png

I use Firefox v.28.0 on Windows 7 environment.
Comment 34 Jonathan Druart 2014-04-03 08:00:12 UTC
(In reply to Paola Rossi from comment #33)
> Your test at comment 32 references to the second moment, where no error is
> shown.

I create the new category after applying patches?

> In the Sandbox 17, I recreated the error, see link:
> 3) http://www.koha.it/local--files/immaginibugs/newcat3-mu.png

Could you please describe, step by step, how do you arrive on this page?
From http://pro.test17.biblibre.com/cgi-bin/koha/admin/authorised_values.pl
I select "Italiano" in languages, I select "newcat-μ" in the dropdown list.
And I see the table with my authorized value, correctly displayed.
The only problem I see is that the dropdown list does not preselect the good value.
What is the character encoding of your browser? (Edition > Character encoding)
Comment 35 Paola Rossi 2014-04-03 13:26:07 UTC
IOO the error depended on the 3 lines of ./admin/authorised_values.pl:

print "Content-Type: text/html ...........

(165,195,227)

because of the lack of charset="utf-8".

Now the 165-th line appears so:
165  print "Content-Type: text/html; charset=\"utf-8\"......

We add charset="utf-8" in these 3 lines in our installation, and now everything's OK. Now the current category is correctly shown in the dropdown list, even if its name contains some UTF-8 chars. So now the dropdown list preselects the good value.

The error appeared only with Firefox 28.0; it didn't appear with Explorer v.10 nor with Chrome v.33.

With Zeno some lines like those are similar in others cgi. See f.i.:
<http://paste.koha-community.org/204>
Comment 36 Jonathan Druart 2014-04-03 13:33:15 UTC
(In reply to Paola Rossi from comment #35)
> IOO the error depended on the 3 lines of ./admin/authorised_values.pl:
> 
> print "Content-Type: text/html ...........
> 
> (165,195,227)
> 
> because of the lack of charset="utf-8".

Great! I am going to provide a followup for that!
Comment 37 Jonathan Druart 2014-04-03 14:24:13 UTC Comment hidden (obsolete)
Comment 38 Jacek Ablewicz 2014-04-04 10:09:58 UTC
Regarding problem described in comment #13 (this issue is not yet fixed in current version): it seems to be caused by / related to '|url' TT filter usage. In OPAC prog theme facets links are not affected; difference being that includes/opac-facets.inc in prog is using (a lot of) |html instead of |url.

Problems with sorting criteria change/link not working are occurring in OPAC bootstrap & prog themes and in the staff searches too. Probably the same underlying cause.
Comment 39 Paola Rossi 2014-04-04 13:49:32 UTC
(In reply to Jonathan Druart from comment #37)
> Created attachment 26794 [details] [review] [review]
> Bug 11944: Some refresh don't indicate the charset to use
> 
> This patch replaces occurrences of META HTTP-EQUIV=Refresh with a
> redirection.
> 
> Note:
>  The list of modified files has been got with the following grep:
>   git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator
> | grep -v \.js
> 
> Test plan: [...]

All the 8 tests are OK.
Comment 40 Jacek Ablewicz 2014-04-05 18:58:03 UTC
So here and there we uri-escape some strings twice (mostly various search query parts, plus some more):

- directly, by uri_escape_utf8() followed by '|url' or '|uri' TT filter,
- indirectly, by using already uri-escaped string in <input value="..."> which will trigger 2nd uri-escape in browser

The other way around, they are uri-unescaped twice too: first in CGI.pm and 2nd time directly by using uri_unescape in Search.pm etc. (including some places when they rather shouldn't; funny things might happen when searching for something with '%' in it..).
But what's more important, delayed 2nd uri_unescape() would prevent those strings from being promoted to 'wide character' despite CGI qw(-utf8). This terribly dirty fix seems to resolve both problems mentioned in comment #13 (in OPAC search; staff search may need another one):

diff --git a/opac/opac-search.pl b/opac/opac-search.pl
index 7a35085..db0545e 100755
--- a/opac/opac-search.pl
+++ b/opac/opac-search.pl
@@ -387,7 +387,8 @@ if ($indexes[0] && !$indexes[1]) {
 }
 # an operand can be a single term, a phrase, or a complete ccl query
 my @operands = $cgi->param('q');
-@operands = map { uri_unescape($_) } @operands;
+## Not a proper fix, just proof-of-concept!!
+@operands = map { /%[a-f\d]{2}/i? Encode::decode('UTF-8', uri_unescape($_)): $_; } @operands;
Comment 41 Paola Rossi 2014-04-07 12:43:10 UTC
New-ing or updating a patron, a password like this one:

実provo

causes the error:

Software error:
input must contain only octets at /usr/lib/perl5/Crypt/Eksblowfish/Bcrypt.pm line 84.
For help, please send mail to the webmaster (webmaster@linux-test), giving this error message and the time and date of the error. 

Against master, the error doesn't appear.

NB. From Koha devel, from M.de.Rooy@rijksmuseum.nl:

Quoting from a previous try to get UTF8- moving, report 9951:

[[ A nice one to start with:
opac-password.pl: The password may contain diacritics. It should be decoded. But this needs a dive into C4/Auth.pm also (and probably in staff too).
And we still have the problem with old incorrect hashes in the table. If you would have an old password with diacritics, you would still like to login.
This is not just adding a flag to CGI, but a report on its own. So I leave that for now; these hashes still work   ]]

How would you suggest to resolve the old passwords given that Koha and used perl libs would encode/decode properly ?

No reason not to support it, but it needs some more thought.. 
-----------------------------------------------------------------
So I pass the status to "Failed QA".
Comment 42 Jacek Ablewicz 2014-04-07 13:29:14 UTC
(In reply to Paola Rossi from comment #41)

> input must contain only octets at /usr/lib/perl5/Crypt/Eksblowfish/Bcrypt.pm
> line 84.

Hm, we can't realy expect Bcrypt to handle perl wide-characters reliably, those hashing algorithms were designed to deal with octets/bytes only..

In particular, if we want all hashes allready stored in the database still to work, the only reasonable solution IMO will be to hash passwords in utf-8 encoded form - and preferably to keep them in as much unaltered form as possible before hashing. Right now we get password params already decoded thanks to CGI qw( -utf8); I guess we can [re]encode them before hashing, but there is no guarantee result string will be allways the same after utf-8 encode(decode()).

BTW, is there some way to fetch on demand a specific parameter from CGI.pm in non-decoded form, i.e. bypassing en effect of qw( -uft8)?
Comment 43 Jonathan Druart 2014-04-07 15:11:22 UTC Comment hidden (obsolete)
Comment 44 Jonathan Druart 2014-04-07 15:11:29 UTC Comment hidden (obsolete)
Comment 45 Jonathan Druart 2014-04-07 15:11:34 UTC Comment hidden (obsolete)
Comment 46 Jonathan Druart 2014-04-07 15:11:39 UTC Comment hidden (obsolete)
Comment 47 Jonathan Druart 2014-04-07 15:11:45 UTC Comment hidden (obsolete)
Comment 48 Jonathan Druart 2014-04-07 15:11:50 UTC Comment hidden (obsolete)
Comment 49 Jonathan Druart 2014-04-07 15:11:55 UTC Comment hidden (obsolete)
Comment 50 Jonathan Druart 2014-04-07 15:12:00 UTC Comment hidden (obsolete)
Comment 51 Jonathan Druart 2014-04-07 15:12:05 UTC Comment hidden (obsolete)
Comment 52 Jonathan Druart 2014-04-07 15:12:10 UTC Comment hidden (obsolete)
Comment 53 Jonathan Druart 2014-04-07 15:12:15 UTC Comment hidden (obsolete)
Comment 54 Jonathan Druart 2014-04-07 15:12:20 UTC Comment hidden (obsolete)
Comment 55 Jonathan Druart 2014-04-07 15:12:25 UTC Comment hidden (obsolete)
Comment 56 Jonathan Druart 2014-04-07 15:12:31 UTC Comment hidden (obsolete)
Comment 57 Jonathan Druart 2014-04-07 15:12:36 UTC Comment hidden (obsolete)
Comment 58 Jonathan Druart 2014-04-07 15:20:09 UTC
(In reply to Paola Rossi from comment #41)
> New-ing or updating a patron, a password like this one:
> 
> 実provo
> 
> causes the error:

This should be fixed by patch "Bug 11944: Authentication"

(In reply to Jacek Ablewicz from comment #40)

It seems I managed to fix that in patch "Bug 11944: Facets and resort".
But I'm not really sure...

Last patch set fix some problems in previous patches: 
  Encode::decode does not work on MARC::Record (of course!)
  Encode::decode returns the decoded value (and does not modify the variable passed in parameter).

I don't reproduce the "importing" error any longer, it seems fixed.

Please add known bugs on the wiki page: http://wiki.koha-community.org/wiki/Handling_UTF8_in_development#Known_bugs
Comment 59 Paola Rossi 2014-04-08 10:06:48 UTC
Today I've installed master 3.15.00.027
I try to apply the patches.
At the 6-th patch there a conflict, perhaps due to the rebasing:

Applying: Bug 11944: use CGI( -utf8 ) everywhere
Using index info to reconstruct a base tree...
<stdin>:2009: trailing whitespace.
use CGI qw ( -utf8 ); 
warning: 1 line adds whitespace errors.
Falling back to patching base and 3-way merge...
Auto-merging tools/koha-news.pl
CONFLICT (content): Merge conflict in tools/koha-news.pl
Auto-merging opac/opac-main.pl
CONFLICT (content): Merge conflict in opac/opac-main.pl
Auto-merging mainpage.pl
CONFLICT (content): Merge conflict in mainpage.pl
Failed to merge in the changes.
Patch failed at 0001 Bug 11944: use CGI( -utf8 ) everywhere
------------------
So I pass the patch to "Patch doesn't apply" status.
Comment 60 Jonathan Druart 2014-04-08 10:24:09 UTC Comment hidden (obsolete)
Comment 61 Jonathan Druart 2014-04-08 10:25:18 UTC
Last patch resolves conflicts with bug 7567.
Comment 62 Paola Rossi 2014-04-09 12:16:31 UTC
In ACQ, I create a fund whose name is "fund1 μμμ" for a budget.
I order a biblio, and I receive the order.
In the "Receipt summary", in the table of the "Already received", I see:
1.A) in the 8-th column "Fund", the name of the fund with the right chars;
1.B) in the 7-th column "Quantity", at the "Subtotal for", the name of the fund with bad chars:
"Subtotal for fund1 μμμ".

-----------------------------------------------------------------
So I pass the status to "Failed QA".
Comment 63 Jonathan Druart 2014-04-09 13:25:58 UTC Comment hidden (obsolete)
Comment 64 Paola Rossi 2014-04-10 13:44:33 UTC
I add a new UniMARC greek authority, i.e.:
....
200$aΜαρωνίτης,
200$bΔ. Ν.
....

The db contains the right chars; from export of the db:
...<subfield code=\"a\">Μαρωνίτης,</subfield>\n    <subfield code=\"b\">Δ. Ν.</subfield>....

When I see the new authority on:
cgi-bin/koha/authorities/detail.pl?authid=1

koha shows tag 200 with the wrong chars, such as:

a Entry Element ÎαÏÏνίÏηÏ,
b Part of Name Other t Î. Î. 

Nel source:
....
                <input type="text"
                        id="tag_200_subfield_a_818751_534076"
                        name="tag_200_subfield_a_818751_534076"
                        value="ÎαÏÏνίÏηÏ,"
                        tabindex="1"
                        size="67"
                        maxlength="9999"
                        class="input_marceditor"
                />
....etc.

These wrong chars are shown in FF, Chrome and Explorer.
-----------------------------------------------------------
So I pass the status to "Failed QA".
Comment 65 Paola Rossi 2014-04-10 15:55:10 UTC
The problem in comment 64 about authorities in koha UniMARC doesn't occur in koha MARC21.
Perhaps the wrong detail occured because of the bad sql file for UniMARC in English set up.
So the status is "Needs Signoff" again.
Comment 66 Paola Rossi 2014-04-11 13:02:56 UTC
In Serials, I create a new numbering pattern whose name is:
new実 numbering pattern

In this pattern I set this Label:
実label

When I select this new pattern for a subscription, the label shown has bad chars:

å®label

I go on and select "Show advanced pattern". The red "Pattern name" shown has bad chars:

newå® numbering pattern
-----------------------------------------------------------
So I pass the status to "Failed QA".
Comment 67 Jonathan Druart 2014-04-11 13:47:45 UTC Comment hidden (obsolete)
Comment 68 Paola Rossi 2014-04-11 15:02:07 UTC
I apply the last "Fix encoding issue" patch. No bad chars are seen,

But, if I use f.i. the "frequency" 1/day, in the "Advanced prediction pattern" shown table "null" is shown in some cells.
In sandbox 17 no "null" is shown in these cells.

IMO this could be a bug, and so I pass the patch to "Failed QA" status.
Comment 69 Jonathan Druart 2014-04-14 09:57:53 UTC Comment hidden (obsolete)
Comment 70 Paola Rossi 2014-04-14 11:02:18 UTC
I create a list whose name is:
学協会. μμ

I add some biblios to the list [the titles of the added biblios contain greek and japanese chars].

I try to "Send list" to my email.
Either I give over-255 chars in the "Comment" of the mail, or I don't give any over-255 char in the "Comment" of the mail, I see this abnormal termination:

Software error:
Wide character in subroutine entry at /var/root-koha/bug-11944/virtualshelves/sendshelf.pl line 134.

I use the 17 sandbox. I create a list whose name is:
学協会. μμ
I add some biblios to the list [the titles of the added biblios contain greek and japanese chars].

Either I give over-255 chars in the "Comment" of the mail, or I don't give any over-255 char in the "Comment" of the mail, the answer of koha is right:

Messaggio inviato
La lista è stata mandata a: paola.rossi@cineca.it
Chiudi la finestra

So I receive the expected mail. The received file is OK with right chars, while the subject of the mail is the string "no subject". 

So I pass the patch to "Failed QA" status.

NB. In the messagge:
La lista è stata mandata a: ...
[i.e. the list has been sent to ...]
the name of the list is not shown, whilest I might select among a lot of lists of mine. So koha gives no feedback about which list has been sent, and I can't see the list name on this page.
Comment 71 Jonathan Druart 2014-04-14 14:37:56 UTC Comment hidden (obsolete)
Comment 72 Paola Rossi 2014-04-15 15:00:08 UTC
1) About sending (=Envoyer) Basket/Cart:
Basket/Cart is sent OK:
- either on intranet or on OPAC, 
- with over-255 chars either in the content of the mail (i.e. Commentaire) or in the content of the attached file (i.e. biblios titles), 
- by English, by French and by Italian

But:
the French message for cart shows bad chars:
Bonjour,
   113 vous a envoy� un panier de r�f�rences extraites de notre
   catalogue.
   Notez que le fichier attach� est un fichier MARC, qui peut �tre
   import� dans un logiciel de gestion de bibliographie tel que EndNote,
   Reference Manager ou ProCite.
the Italian message for cart shows bad chars:
Ciao,
   113 ti ha inviato un carrello dal nostro catalogo online.
   Il file in allegato � fatto di record MARC che possono essere
   importati in un software bibliografico come EndNote, Reference
   Manager, ProCite, ecc.
------------------------------------------------------------------------------
2) About sending (=Envoyer) Shelf/List:
on OPAC, list is sent OK:
- with over-255 chars either in the content of the mail (i.e. Commentaire) or in the content of the attached file (i.e. biblios titles), 
- by English, by French and by Italian.

On intranet: 
-- French doesn't send any list [with or without over-255 chars (in biblios titles and in the list name)];
-- Italian doesn't send any list [with or without over-255 chars (in biblios titles and in the list name)]; [i.e. Italian same as French]
-- English sends the list [with or without over-255 chars (in biblios titles and in the list name)]; the attached file is ok, while the content of the mail shows bad chars over-127 (biblionumber 1 in sand box 17:

La Recherche å­¦åä¼. μμ
by
Published by: Stépahanie KHEMIS, 57
--------------------------------------------------------------
The attached files are all OK.
--------------------------------------------------------------
So I pass the patch to "Failed QA" status.
Comment 73 Jonathan Druart 2014-04-15 15:46:53 UTC
(In reply to Paola Rossi from comment #72)
Arg, it works for me. Which email client do you use?
Comment 74 Zeno Tajoli 2014-04-15 15:50:18 UTC
I and Paola we use Thunderbird 24.4.0
Comment 75 Jonathan Druart 2014-04-16 08:30:24 UTC Comment hidden (obsolete)
Comment 76 Jonathan Druart 2014-04-16 08:30:39 UTC Comment hidden (obsolete)
Comment 77 Jonathan Druart 2014-04-16 08:31:40 UTC
This patch set now depends on bug 11248.
Comment 78 Jonathan Druart 2014-04-16 08:37:23 UTC
(In reply to Zeno Tajoli from comment #74)
> I and Paola we use Thunderbird 24.4.0

I reproduced using icedove.
The "header" was not encoded. It should work now.
Comment 79 Jonathan Druart 2014-04-16 09:15:09 UTC Comment hidden (obsolete)
Comment 80 Paola Rossi 2014-04-16 12:26:14 UTC
I apply against master 3.15.00.030 and at the 16-th patch:

Applying: Bug 11944: Fix encoding issue on adding a subscription
fatal: sha1 information is lacking or useless (serials/subscription-frequency.pl).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 11944: Fix encoding issue on adding a subscription
-----------------------------------------------------------------------------
So I pass the patch to "Patch doesn't apply" status.
Comment 81 Jonathan Druart 2014-04-16 12:47:55 UTC Comment hidden (obsolete)
Comment 82 Jonathan Druart 2014-04-16 12:48:04 UTC Comment hidden (obsolete)
Comment 83 Jonathan Druart 2014-04-16 12:48:10 UTC Comment hidden (obsolete)
Comment 84 Jonathan Druart 2014-04-16 12:48:36 UTC Comment hidden (obsolete)
Comment 85 Paola Rossi 2014-04-16 14:04:03 UTC
I apply against master 3.15.00.030 and at the 17-th patch:

Applying: Bug 11944: use CGI( -utf8 ) everywhere
fatal: sha1 information is lacking or useless (opac/opac-sendshelf.pl).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 11944: use CGI( -utf8 ) everywhere

-----------------------------------------------------------------------------
So I pass the patch to "Patch doesn't apply" status.
Comment 86 Jonathan Druart 2014-04-16 14:12:31 UTC Comment hidden (obsolete)
Comment 87 Jonathan Druart 2014-04-16 14:12:36 UTC Comment hidden (obsolete)
Comment 88 Jonathan Druart 2014-04-16 14:12:54 UTC Comment hidden (obsolete)
Comment 89 Jonathan Druart 2014-04-16 14:13:02 UTC Comment hidden (obsolete)
Comment 90 Jonathan Druart 2014-04-16 14:13:10 UTC Comment hidden (obsolete)
Comment 91 Jonathan Druart 2014-04-16 14:13:16 UTC Comment hidden (obsolete)
Comment 92 Jonathan Druart 2014-04-16 14:13:21 UTC Comment hidden (obsolete)
Comment 93 Jonathan Druart 2014-04-16 14:13:26 UTC Comment hidden (obsolete)
Comment 94 Jonathan Druart 2014-04-16 14:13:30 UTC Comment hidden (obsolete)
Comment 95 Jonathan Druart 2014-04-16 14:13:35 UTC Comment hidden (obsolete)
Comment 96 Jonathan Druart 2014-04-16 14:13:41 UTC Comment hidden (obsolete)
Comment 97 Jonathan Druart 2014-04-16 14:13:46 UTC Comment hidden (obsolete)
Comment 98 Jonathan Druart 2014-04-16 14:13:51 UTC Comment hidden (obsolete)
Comment 99 Jonathan Druart 2014-04-16 14:13:56 UTC Comment hidden (obsolete)
Comment 100 Jonathan Druart 2014-04-16 14:14:04 UTC Comment hidden (obsolete)
Comment 101 Jonathan Druart 2014-04-16 14:14:10 UTC Comment hidden (obsolete)
Comment 102 Jonathan Druart 2014-04-16 14:14:16 UTC Comment hidden (obsolete)
Comment 103 Jonathan Druart 2014-04-16 14:14:21 UTC Comment hidden (obsolete)
Comment 104 Jonathan Druart 2014-04-16 14:14:27 UTC Comment hidden (obsolete)
Comment 105 Jonathan Druart 2014-04-16 14:15:05 UTC
I resubmit all 19 patches.
Comment 106 Paola Rossi 2014-04-16 15:14:15 UTC
I'm sorry, but I see the problem again with the new patches.
The problem is still on the 17-th patch, about the same pl:

Applying: Bug 11944: use CGI( -utf8 ) everywhere
fatal: sha1 information is lacking or useless (opac/opac-sendshelf.pl).
Repository lacks necessary blobs to fall back on 3-way merge.
Cannot fall back to three-way merge.
Patch failed at 0001 Bug 11944: use CGI( -utf8 ) everywhere
-----------------------------------------------------------------------------
This is the opac/opac-sendshelf.pl from 20-th line on:

use strict;
use warnings;
 
use CGI;
use Encode;
use Carp;
....
-----------------------------------------------------------------------------
whilest these are the lines searched by the 17-th patch:

-use CGI;
-use Encode qw(decode encode);

-----------------------------------------------------------------------------
So I pass the patch to "Patch doesn't apply" status.
Comment 107 Jonathan Druart 2014-04-16 15:24:45 UTC
(In reply to Paola Rossi from comment #106)

It should work. Did you apply patches from bug 11248 before?
Comment 108 Paola Rossi 2014-04-16 16:26:55 UTC
(In reply to Jonathan Druart from comment #107)
> (In reply to Paola Rossi from comment #106)
> 
> It should work. Did you apply patches from bug 11248 before?

That was the error!

Now I've started applying the 3 patches of 11248, and so I can apply all the 22 patches.
So I pass the patch to "Needs Signoff" status.

[Until May the 5-th I'll rarely be on bugzilla.]
Comment 109 Paola Rossi 2014-04-24 16:40:37 UTC
I test against master 3.15.00.036.
[Preference UseICU set to "Using"]

I've tested ACQ, Serials and the Advanced Search module. 

IMO the patch is OK about these 3 modules.

So I pass the patch to "Signed Off" status.

-----------------------------------------------------------------------------
Some doc about advanced search test.

I create some [fictious] authorities (personal name' author, topical term' subject) with Japanese (such as 本学会誌), Greek (Σῖγμασς), Arabic (مكن إجراء البحث من), and some other (ČčĕīōŠ) chars. 
Then I fill the 700$a and 650$a of some biblios with them. Titles of the biblios have those chars too.

Some fields are like the following ones (1 alphabet's chars only):
يمكن إجراء البحث من
Σῖγμα, σῖγςμας
本, 学会誌

whilest other fields are mixture (!) of some alphabets' chars, such as:
Čajko author Ibn Gĕbīrōl, Šĕlōmōh ben Yĕhudāh يمكن إجراء البحث من
Alcock 実 μσῖ, Ian
Č topic mu μμ Gĕbīrōl, Šĕlōmōh ben Yĕhudāh, Petr Ilʹič

I think the mixture fields are perhaps not real cases: anyway I've checked the koha's behaviour on these hypothetical cases too.

Searching is OK (keyword, keyword as phrase, Author, Author as phrase, Personal name, Personal name as phrase, Subject, Subject as phrase). 
Facets are created well. The link from a facet is OK (the answer is correct), with the exception of NB1.

NB1. The mixed fields with 2 [or more] adjacent Japanese chars are well listed in facets, but these facets have bad links, selecting erroneuosly nothing;
i.e. these kind of facets find nothing [because of the two adjacent Japanese chars inside]:
No results match your search for 'kw,wrdl: ...' with limit(s): 'au:Σῖγμαprovo実会 mu μμμ, σῖγςμας'

NB2. If I search for 'Č' in :

advanced search, f.i. by "Keyword" field, [in ADVANCED SEARCH]
search authorities by "Contains" operator, [in AUTHORITIES]
search authorities by Tag editor, f.i. by "Main Entry contains" [in CATALOG]

the answer is the same as the answer against searching for "c". Perhaps this is the expected behaviour.
-----------------------------------------------------------------------------
Comment 110 Jonathan Druart 2014-04-25 09:45:15 UTC Comment hidden (obsolete)
Comment 111 Jonathan Druart 2014-04-25 09:45:19 UTC Comment hidden (obsolete)
Comment 112 Jonathan Druart 2014-04-25 09:45:24 UTC Comment hidden (obsolete)
Comment 113 Jonathan Druart 2014-04-25 09:45:37 UTC Comment hidden (obsolete)
Comment 114 Jonathan Druart 2014-04-25 09:45:41 UTC Comment hidden (obsolete)
Comment 115 Jonathan Druart 2014-04-25 09:45:46 UTC Comment hidden (obsolete)
Comment 116 Jonathan Druart 2014-04-25 09:45:50 UTC Comment hidden (obsolete)
Comment 117 Jonathan Druart 2014-04-25 09:45:54 UTC Comment hidden (obsolete)
Comment 118 Jonathan Druart 2014-04-25 09:45:58 UTC Comment hidden (obsolete)
Comment 119 Jonathan Druart 2014-04-25 09:46:03 UTC Comment hidden (obsolete)
Comment 120 Jonathan Druart 2014-04-25 09:46:07 UTC Comment hidden (obsolete)
Comment 121 Jonathan Druart 2014-04-25 09:46:11 UTC Comment hidden (obsolete)
Comment 122 Jonathan Druart 2014-04-25 09:46:15 UTC Comment hidden (obsolete)
Comment 123 Jonathan Druart 2014-04-25 09:46:20 UTC Comment hidden (obsolete)
Comment 124 Jonathan Druart 2014-04-25 09:46:24 UTC Comment hidden (obsolete)
Comment 125 Jonathan Druart 2014-04-25 09:46:28 UTC Comment hidden (obsolete)
Comment 126 Jonathan Druart 2014-04-25 09:46:33 UTC Comment hidden (obsolete)
Comment 127 Jonathan Druart 2014-04-25 09:46:37 UTC Comment hidden (obsolete)
Comment 128 Jonathan Druart 2014-04-25 09:46:41 UTC Comment hidden (obsolete)
Comment 129 Jonathan Druart 2014-04-25 09:51:48 UTC
(In reply to Paola Rossi from comment #109)
> So I pass the patch to "Signed Off" status.

Thanks Paola!
I added your signoff to the patch set.
I switch the status back to Needs Signoff for a second signoff.
Comment 130 Galen Charlton 2014-04-29 14:48:22 UTC
Marking as candidate for 3.16.0.
Comment 131 Bernardo Gonzalez Kriegel 2014-05-01 16:18:20 UTC Comment hidden (obsolete)
Comment 132 Bernardo Gonzalez Kriegel 2014-05-01 16:18:30 UTC Comment hidden (obsolete)
Comment 133 Bernardo Gonzalez Kriegel 2014-05-01 16:18:42 UTC Comment hidden (obsolete)
Comment 134 Bernardo Gonzalez Kriegel 2014-05-01 16:18:52 UTC Comment hidden (obsolete)
Comment 135 Bernardo Gonzalez Kriegel 2014-05-01 16:19:02 UTC Comment hidden (obsolete)
Comment 136 Bernardo Gonzalez Kriegel 2014-05-01 16:24:39 UTC
I'm in the process of sign, but have problems to upload some patches with git-bz.
I'm out for a moment, will finish soon
Comment 137 Bernardo Gonzalez Kriegel 2014-05-01 16:35:20 UTC Comment hidden (obsolete)
Comment 138 Bernardo Gonzalez Kriegel 2014-05-01 16:35:31 UTC Comment hidden (obsolete)
Comment 139 Bernardo Gonzalez Kriegel 2014-05-01 16:35:43 UTC Comment hidden (obsolete)
Comment 140 Bernardo Gonzalez Kriegel 2014-05-01 16:35:54 UTC Comment hidden (obsolete)
Comment 141 Bernardo Gonzalez Kriegel 2014-05-01 16:36:06 UTC Comment hidden (obsolete)
Comment 142 Bernardo Gonzalez Kriegel 2014-05-01 16:36:19 UTC Comment hidden (obsolete)
Comment 143 Bernardo Gonzalez Kriegel 2014-05-01 16:36:30 UTC Comment hidden (obsolete)
Comment 144 Bernardo Gonzalez Kriegel 2014-05-01 16:36:41 UTC Comment hidden (obsolete)
Comment 145 Bernardo Gonzalez Kriegel 2014-05-01 16:43:33 UTC Comment hidden (obsolete)
Comment 146 Bernardo Gonzalez Kriegel 2014-05-01 16:43:45 UTC Comment hidden (obsolete)
Comment 147 Bernardo Gonzalez Kriegel 2014-05-01 16:44:01 UTC Comment hidden (obsolete)
Comment 148 Bernardo Gonzalez Kriegel 2014-05-01 16:44:16 UTC Comment hidden (obsolete)
Comment 149 Bernardo Gonzalez Kriegel 2014-05-01 16:44:35 UTC Comment hidden (obsolete)
Comment 150 Bernardo Gonzalez Kriegel 2014-05-01 16:49:36 UTC Comment hidden (obsolete)
Comment 151 Bernardo Gonzalez Kriegel 2014-05-01 16:50:34 UTC
Sorry, I need to rewrite some text on commit messages.
But is finished
Comment 152 Galen Charlton 2014-05-05 21:13:54 UTC
On reflection -- I'm not going to bypass QA on this one, given its size.  Remove the 3.16 candidate keyword.
Comment 153 Zeno Tajoli 2014-05-06 04:46:26 UTC
Do you think it is still possible to insert it ?
In fact it doesn't change template strings so it is not blocked by string freeze.
Comment 154 Jonathan Druart 2014-05-06 07:15:23 UTC
I think it would be wise not to push it into master before the release. But it would be great to have the patches into 3.17 asap, in order to catch remaining bugs by the developer's team.
Comment 155 Jonathan Druart 2014-05-08 10:17:39 UTC Comment hidden (obsolete)
Comment 156 Jonathan Druart 2014-05-08 10:17:44 UTC Comment hidden (obsolete)
Comment 157 Jonathan Druart 2014-05-08 10:17:48 UTC Comment hidden (obsolete)
Comment 158 Jonathan Druart 2014-05-08 10:17:52 UTC Comment hidden (obsolete)
Comment 159 Jonathan Druart 2014-05-08 10:17:57 UTC Comment hidden (obsolete)
Comment 160 Jonathan Druart 2014-05-08 10:18:01 UTC Comment hidden (obsolete)
Comment 161 Jonathan Druart 2014-05-08 10:18:05 UTC Comment hidden (obsolete)
Comment 162 Jonathan Druart 2014-05-08 10:18:10 UTC Comment hidden (obsolete)
Comment 163 Jonathan Druart 2014-05-08 10:18:14 UTC Comment hidden (obsolete)
Comment 164 Jonathan Druart 2014-05-08 10:18:18 UTC Comment hidden (obsolete)
Comment 165 Jonathan Druart 2014-05-08 10:18:23 UTC Comment hidden (obsolete)
Comment 166 Jonathan Druart 2014-05-08 10:18:27 UTC Comment hidden (obsolete)
Comment 167 Jonathan Druart 2014-05-08 10:18:31 UTC Comment hidden (obsolete)
Comment 168 Jonathan Druart 2014-05-08 10:18:35 UTC Comment hidden (obsolete)
Comment 169 Jonathan Druart 2014-05-08 10:18:39 UTC Comment hidden (obsolete)
Comment 170 Jonathan Druart 2014-05-08 10:18:44 UTC Comment hidden (obsolete)
Comment 171 Jonathan Druart 2014-05-08 10:18:49 UTC Comment hidden (obsolete)
Comment 172 Jonathan Druart 2014-05-08 10:18:53 UTC Comment hidden (obsolete)
Comment 173 Jonathan Druart 2014-05-08 10:18:57 UTC Comment hidden (obsolete)
Comment 174 Jonathan Druart 2014-05-08 10:20:30 UTC
I rebased my local branch against master, added the Bernardo'SO to the patches and reuploaded them. Original commit messages are present.
Comment 175 Jonathan Druart 2014-05-08 10:50:51 UTC Comment hidden (obsolete)
Comment 176 Jonathan Druart 2014-05-08 11:02:00 UTC Comment hidden (obsolete)
Comment 177 Marcel de Rooy 2014-05-08 11:04:45 UTC
(In reply to Jonathan Druart from comment #176)
> However, it is useless to decode column names.

I am not sure about that..
Comment 178 Benjamin Rokseth 2014-05-08 11:48:53 UTC
Great to see this thorough work on utf8!
We have done a lot of manual corrections on .pref files previously.

However, applying these patches on a clean master and running

translate install nb-NO

now leads to EVERY bit of koha becoming double encoded...

example:
 Advanced search | Authority search | Tag cloud 
becomes:
 Avansert søk | Autoritetssøk | Taggsky 


is there something I'm missing? I've tried swedish as well, with same result.
Comment 179 Jonathan Druart 2014-05-08 12:20:01 UTC
(In reply to Benjamin Rokseth from comment #178)
> example:
>  Advanced search | Authority search | Tag cloud 
> becomes:
>  Avansert søk | Autoritetssøk | Taggsky 
> 
> 
> is there something I'm missing? I've tried swedish as well, with same result.

I just tried and it seems to work:
    Avansert søk     Course reserves     Autoritetssøk     Taggsky     Emnesky     Mest populære

At the OPAC, on the main page.
Comment 180 Jonathan Druart 2014-05-08 12:22:26 UTC
I just found a bug with the search, it's completely broken.
Comment 181 Jonathan Druart 2014-05-08 14:07:34 UTC Comment hidden (obsolete)
Comment 182 Jonathan Druart 2014-05-08 14:11:17 UTC
(In reply to Jonathan Druart from comment #181)
> Created attachment 28097 [details] [review] [review]
> Bug 11944: Facets and resort

I need more testing here. This patch does not fix everything but I didn't manage to list what is broken or not.

I need some expert too: what is the best way to handle the uri_escape/unescape and the uri, url, html TT filters?
There are a lot of all of that in include files, results.tt, search.pl and Seach.pm...
Comment 183 Marcel de Rooy 2014-05-12 08:55:24 UTC
(In reply to Jonathan Druart from comment #182)
> (In reply to Jonathan Druart from comment #181)
> I need some expert too: what is the best way to handle the
> uri_escape/unescape and the uri, url, html TT filters?
> There are a lot of all of that in include files, results.tt, search.pl and
> Seach.pm...

Looking at unescape:

uri_escape_utf8 encodes data before escaping, so expects perl internal format (what you should normally have..)
uri_escape and uri_unescape operate on byte level; so you should normally expect to decode their output in your program.
Also note that CGI normally handles escaping/unescaping for params. So it seems that some unescaping in catalogue/search.pl and opac-search.pl is useless.

The unescape in members/paycollect.pl is not decoded. Could be a problem.
In cataloguing/additem a cookie is unescaped and thawed. Should be checked.

This one is confusing (opac-search):
my @query_inputs = _input_cgi_parse($query_cgi);
$template->param ( QUERY_INPUTS => \@query_inputs );
$query_cgi comes from buildQuery and contains uri_escape_utf8ed string (so: encoded data)
_input_cgi_parse runs a part through uri_unescape, so still encoded.
we pass it to the template in encoded form: Does it get mixed up with decoded data here?

C4/Search/History:
decode_json( uri_unescape( $session->param('search_history') ) )
$session is a CGI Session object: How does it handle utf8?
uri_unescape operates on byte level
decode_json expects an encoded string
Should be checked. Not sure if the output is encoded or decoded in perl.
Comment 184 Magnus Enger 2014-05-13 08:09:33 UTC
(In reply to Benjamin Rokseth from comment #178)
> However, applying these patches on a clean master and running
> translate install nb-NO
> now leads to EVERY bit of koha becoming double encoded...

I worked with Benjamin last week to try and recreate this, but we did not find a solution. We were both on Ubuntu 14.04, with the same setup and the same versions of important modules. Benjamin's templates were broken and mine were not. The only difference was that I had cloned the code from git.koha-community.org and Benjamin had cloned from some mirror (due to firewall restrictions).
Comment 185 Mark Tompsett 2014-05-13 20:19:43 UTC
Comment on attachment 28062 [details] [review]
Bug 11944: use CGI( -utf8 ) everywhere

Review of attachment 28062 [details] [review]:
-----------------------------------------------------------------

::: members/guarantor_search.pl
@@ +24,4 @@
>  #use warnings; FIXME - Bug 2505
>  use C4::Auth;
>  use C4::Output;
> +use CGI qw ( -utf8 );

We have a merge conflict against master now. :(
Comment 186 Mark Tompsett 2014-05-13 20:20:55 UTC
Comment on attachment 28063 [details] [review]
Bug 11944: Fix encoding on sending emails

Review of attachment 28063 [details] [review]:
-----------------------------------------------------------------

This patch has merge conflicts. :(
Comment 187 Jonathan Druart 2014-05-14 08:35:21 UTC Comment hidden (obsolete)
Comment 188 Jonathan Druart 2014-05-14 08:35:27 UTC Comment hidden (obsolete)
Comment 189 Jonathan Druart 2014-05-14 08:36:30 UTC Comment hidden (obsolete)
Comment 190 Jonathan Druart 2014-05-14 08:36:36 UTC Comment hidden (obsolete)
Comment 191 Jonathan Druart 2014-05-14 08:36:44 UTC Comment hidden (obsolete)
Comment 192 Jonathan Druart 2014-05-14 08:36:51 UTC Comment hidden (obsolete)
Comment 193 Jonathan Druart 2014-05-14 08:37:36 UTC
(In reply to M. Tompsett from comment #186)
> Comment on attachment 28063 [details] [review] [review]
> This patch has merge conflicts. :(

Last 6 patches rebased.
Comment 194 Petter Goksøyr Åsen 2014-05-14 09:17:28 UTC
Overall, looking pretty good with Norwegian characters now, thank you for your efforts!

So far I only found two issues:
material-types are double encoded in the item circulation alerts table:

/cgi-bin/koha/admin/item_circulation_alerts.pl

which made me want to check material types at

/cgi-bin/koha/admin/itemtypes.pl

but it fails with

Template process failed: plugin error - EncodeUTF8: plugin not found at /usr/local/src/kohaclone/C4/Templates.pm line 124.
Comment 195 Jonathan Druart 2014-05-14 09:26:22 UTC Comment hidden (obsolete)
Comment 196 Magnus Enger 2014-05-14 09:29:30 UTC
(In reply to Petter Goksøyr Åsen from comment #194)
> So far I only found two issues:
> material-types are double encoded in the item circulation alerts table:
> 
> /cgi-bin/koha/admin/item_circulation_alerts.pl
> 
> which made me want to check material types at
> 
> /cgi-bin/koha/admin/itemtypes.pl
> 
> but it fails with
> 
> Template process failed: plugin error - EncodeUTF8: plugin not found at
> /usr/local/src/kohaclone/C4/Templates.pm line 124.

I can confirm both of these problems. It looks like the EncodeUTF8 plugin is removed in attachment 28052 [details] [review], but it is still used in a number of places, after the patches on this bug have been applied:

$ grep -rl --exclude=*.po "EncodeUTF8" *
koha-tmpl/intranet-tmpl/prog/en/modules/admin/itemtypes.tt
koha-tmpl/intranet-tmpl/prog/en/modules/circ/renew.tt
koha-tmpl/intranet-tmpl/prog/nb-NO/modules/admin/itemtypes.tt
koha-tmpl/intranet-tmpl/prog/nb-NO/modules/circ/renew.tt
Comment 197 Jonathan Druart 2014-05-14 09:32:07 UTC Comment hidden (obsolete)
Comment 198 Magnus Enger 2014-05-14 09:40:25 UTC Comment hidden (obsolete)
Comment 199 Magnus Enger 2014-05-14 09:41:11 UTC
Comment on attachment 28252 [details] [review]
Bug 11944: (follow-up) Remove all utf8 filter from templates

Uploaded a signed off version of this
Comment 200 Paola Rossi 2014-05-30 09:31:04 UTC
Today I try to apply against master 3.17.00.007

The first patch doesn't apply:
Applying: Bug 11944: Remove all utf8 filter from templates
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging koha-tmpl/opac-tmpl/prog/en/modules/opac-account.tt
CONFLICT (content): Merge conflict in koha-tmpl/opac-tmpl/prog/en/modules/opac-account.tt
Auto-merging Koha/Template/Plugin/Branches.pm
Failed to merge in the changes.
Patch failed at 0001 Bug 11944: Remove all utf8 filter from templates

So I pass the patch to "Patch doesn't apply" status.
Comment 201 Jonathan Druart 2014-06-09 10:37:29 UTC Comment hidden (obsolete)
Comment 202 Jonathan Druart 2014-06-09 10:40:06 UTC
It's really a pain to upload/maintain these patches up-to-date on the bug report. I pushed the patches on a remote branch:
  https://git.biblibre.com/biblibre/kohac/commits/ft/bug_11944

The url of the remote is http://git.biblibre.com/biblibre/kohac.git
Comment 203 Jonathan Druart 2014-06-09 10:45:40 UTC
commit 35884cf951c2aaa7d57daef264be9eca3c2760bc
Author: Jonathan Druart <jonathan.druart@biblibre.com>
Date:   Mon Jun 9 12:44:35 2014 +0200

    Revert "Bug 12279: Diacritics in system preference editor broken"
    
    This reverts commit 2fa2480447a58f9d5d3f7834bcfdcf62a0a95df3.
    
    Conflicts:
        admin/preferences.pl

It fixes regression introduced by bug 12279.
Comment 204 Jonathan Druart 2014-06-12 11:31:14 UTC
Sandboxes 6 and 16 use this branch.
Comment 205 Tomás Cohen Arazi 2014-07-02 19:07:28 UTC
*** Bug 10068 has been marked as a duplicate of this bug. ***
Comment 206 Zeno Tajoli 2014-07-11 08:21:03 UTC
A General Bug Sign off Day only for this bug is planned for 23/July/2014
Comment 207 Jonathan Druart 2014-07-15 10:22:37 UTC
Branch updated against master.
Comment 208 Jonathan Druart 2014-07-23 15:03:22 UTC
TODO: add an updatedb entry to change the charset collate to utf8_unicode_ci.

Somethink like

SET foreign_key_checks = 0;
foreach table
  ALTER TABLE `table` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci;
end
SET foreign_key_checks = 1;
Comment 209 Dobrica Pavlinusic 2014-07-23 16:05:51 UTC Comment hidden (obsolete)
Comment 210 Dobrica Pavlinusic 2014-07-23 16:05:59 UTC Comment hidden (obsolete)
Comment 211 Dobrica Pavlinusic 2014-07-23 16:07:32 UTC Comment hidden (obsolete)
Comment 212 Dobrica Pavlinusic 2014-07-23 16:07:56 UTC Comment hidden (obsolete)
Comment 213 Dobrica Pavlinusic 2014-07-23 16:08:27 UTC Comment hidden (obsolete)
Comment 214 Dobrica Pavlinusic 2014-07-23 16:12:55 UTC Comment hidden (obsolete)
Comment 215 Dobrica Pavlinusic 2014-07-23 16:13:05 UTC Comment hidden (obsolete)
Comment 216 Dobrica Pavlinusic 2014-07-23 16:13:14 UTC Comment hidden (obsolete)
Comment 217 Dobrica Pavlinusic 2014-07-23 16:13:23 UTC Comment hidden (obsolete)
Comment 218 Dobrica Pavlinusic 2014-07-23 16:13:41 UTC Comment hidden (obsolete)
Comment 219 Dobrica Pavlinusic 2014-07-23 16:13:50 UTC Comment hidden (obsolete)
Comment 220 Dobrica Pavlinusic 2014-07-23 16:14:01 UTC Comment hidden (obsolete)
Comment 221 Dobrica Pavlinusic 2014-07-23 16:16:36 UTC Comment hidden (obsolete)
Comment 222 Dobrica Pavlinusic 2014-07-23 16:17:01 UTC Comment hidden (obsolete)
Comment 223 Dobrica Pavlinusic 2014-07-23 16:17:13 UTC Comment hidden (obsolete)
Comment 224 Dobrica Pavlinusic 2014-07-23 16:17:28 UTC Comment hidden (obsolete)
Comment 225 Dobrica Pavlinusic 2014-07-23 16:17:51 UTC Comment hidden (obsolete)
Comment 226 Dobrica Pavlinusic 2014-07-23 16:20:23 UTC Comment hidden (obsolete)
Comment 227 Dobrica Pavlinusic 2014-07-23 16:22:29 UTC Comment hidden (obsolete)
Comment 228 Dobrica Pavlinusic 2014-07-23 16:22:52 UTC Comment hidden (obsolete)
Comment 229 Dobrica Pavlinusic 2014-07-23 16:23:17 UTC Comment hidden (obsolete)
Comment 230 Dobrica Pavlinusic 2014-07-23 16:23:50 UTC Comment hidden (obsolete)
Comment 231 Dobrica Pavlinusic 2014-07-23 16:24:19 UTC Comment hidden (obsolete)
Comment 232 Dobrica Pavlinusic 2014-07-23 16:24:50 UTC Comment hidden (obsolete)
Comment 233 Dobrica Pavlinusic 2014-07-23 16:25:20 UTC Comment hidden (obsolete)
Comment 234 Dobrica Pavlinusic 2014-07-23 16:25:55 UTC Comment hidden (obsolete)
Comment 235 Jonathan Druart 2014-07-24 08:27:53 UTC Comment hidden (obsolete)
Comment 236 Jonathan Druart 2014-07-24 08:33:48 UTC
The branch kohac/ft/bug_11944 is set to master + all patches from this report.
The 2 sandboxes 6 and 16 are up-to-date from this branches.
Comment 237 Marcel de Rooy 2014-07-24 08:43:27 UTC
(In reply to Jonathan Druart from comment #235)
> Created attachment 30012 [details] [review] [review]
> Bug 11944: Convert DB tables to utf8_unicode_ci

You are also doing the same at install time?
Comment 238 Jonathan Druart 2014-07-24 09:10:14 UTC Comment hidden (obsolete)
Comment 239 Jonathan Druart 2014-07-24 09:10:58 UTC
(In reply to M. de Rooy from comment #237)
> (In reply to Jonathan Druart from comment #235)
> > Created attachment 30012 [details] [review] [review] [review]
> > Bug 11944: Convert DB tables to utf8_unicode_ci
> 
> You are also doing the same at install time?

Done in the last patch.
Comment 240 Jonathan Druart 2014-08-18 15:35:39 UTC Comment hidden (obsolete)
Comment 241 Jonathan Druart 2014-08-18 15:39:25 UTC
(In reply to Jonathan Druart from comment #240)
> Created attachment 30945 [details] [review] [review]
> Bug 11944: Fix suggestion at the OPAC for perl v5.10

The last patch fixes the issue raised on bug 12664.
The patch is added to the branch kohac/ft/bug_11944.
Sandboxes 6 and 16 are up-to-date.
Comment 242 Jonathan Druart 2014-08-29 15:44:44 UTC Comment hidden (obsolete)
Comment 243 Jonathan Druart 2014-08-29 15:45:31 UTC
(In reply to Jonathan Druart from comment #242)
> Created attachment 31299 [details] [review] [review]
> Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci

The remote branch does not contain this patch at the moment.
Comment 244 Jonathan Druart 2014-09-01 07:26:54 UTC
Last patch push on kohac/ft/bug_11944.
This branch is up-to-date against master.
Comment 245 Jonathan Druart 2014-09-01 10:05:57 UTC
sandboxes 6 and 16 up-to-date.
Comment 246 Martin Renvoize 2014-09-02 15:43:34 UTC
Agreed in Dev Meeting of 02/09/2014 to set status to Signed off..

Branch still maintained by Jonathan is the most up to date patch set.  We will add Zeno and Paula's signoff attribute at the last minute ;)
Comment 247 Martin Renvoize 2014-09-03 12:28:38 UTC
Comment on attachment 31299 [details] [review]
Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci

Review of attachment 31299 [details] [review]:
-----------------------------------------------------------------

First QA point, as mentioned in the Dev meeting, I believe we should jump straight to utf8mb4 for mysql encoding rather than utf8.

Should be a trivial change to the last patch. I can confirm that ptfs-europe have been running utf8mb4 databases with koha for a number of years without issue. We have never come up against any problems caused by the character number reduction associated with certain datatypes in utf8mb4 mysql.
Comment 248 Martin Renvoize 2014-09-30 06:40:04 UTC
OK, I'm pretty much happy to pass this all now, with the exception of I'd like the utf8_mb4 follow up for the database as commented above.. or at least some further input regarding it. I know Galen supported the notion, so unless anyone has a reason not to do it, then I would like to see that change.

Fantastic work Jonathan, was a long hard slog, but worth it.

As a secondary note, Tomas/Galen, did you get anywhere with writing tests to prevent us getting tied in knots again in the future?
Comment 249 Martin Renvoize 2014-09-30 08:46:00 UTC
Happy to Passed QA. In my extensive testing I could not find any regressions, and the code all comes up to scratch passing qa script and tests all still pass.

We still need the unit tests, but as Tomas/Galen promised to get onto these, I don't see them holding up QA on this part of the proceedings.

Jonathan, IRC Gang and I all chatted about utf8_mb4 and we ended up realising that Squeeze still ships with a mysql version over 4 yeras old and so does not support the _mb4 encoding.  I don't feel there is enough benefit to be had from mb4 to ask Jonathan to add a followup with code to decide encoding based upon distrobution.  (Though I'd love to see this some time, and would suggest opening a followup bug to track it)

Fantastic work all involved!

PS: Only 'passed' in bugzilla status, Jonathan will add the signoff line in the patches for me. This is to 'keep it simple' regarding conflicts and re-basing :)
Comment 250 Jonathan Druart 2014-09-30 08:59:17 UTC
(In reply to Martin Renvoize from comment #249)
> PS: Only 'passed' in bugzilla status, Jonathan will add the signoff line in
> the patches for me. This is to 'keep it simple' regarding conflicts and
> re-basing :)

Done, signed-off-by line added on 29 patches.
Comment 251 Tomás Cohen Arazi 2014-10-23 13:28:45 UTC
C4::Charset::IsStringUTF8ish seems to be failing some tests. My feeling is that utf8::decode should still be used there, but am waiting for Galen's answer (as the one who wrote the piece of code).
Comment 252 Jonathan Druart 2014-12-04 12:18:47 UTC
commit bfc167d532788154db64c65c04d80ef603fa56a5
Author: Tomas Cohen Arazi <tomascohen@gmail.com>
Date:   Wed Nov 5 11:59:38 2014 -0300

    Bug 11944: revert unneeded IsStringUTF8ish behaviour change
    
    Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 253 Jonathan Druart 2014-12-04 12:19:00 UTC
commit 8e4f43aee283869dc78991874ea0ac58cdba8ac0
Author: Jonathan Druart <jonathan.druart@biblibre.com>
Date:   Thu Dec 4 13:13:30 2014 +0100

    Bug 11944: Fix problems introduced by bug 11515
    
    The game continue...
    Patch for bug 11515 introduced regression on this patch set.
    To reproduce: search with utf8 characters at the opac
    Test plan:
    Verify that the issue described on bug 11515 is still fixed and that no
    regression is found at the OPAC.
    
    Note that I am pretty sure this patch is not enough.
Comment 254 Jonathan Druart 2014-12-04 12:23:59 UTC
sandboxes 6 and 16 updated with last patches.
Comment 255 Paola Rossi 2014-12-04 16:12:13 UTC
I've made some tests on the two sandboxes 6 and 16.

The issue described on bug 11515 is still fixed: everything is OK imo.
Comment 256 Tomás Cohen Arazi 2014-12-26 14:19:26 UTC
Guys, can u please take a look at the tests from 13264? It seems that the latest refactoring of the tests yield a situation where the tests don't pass.
Comment 257 Katrin Fischer 2015-01-11 22:04:29 UTC
Tried to apply this for testing the tests, but it won't apply. The branch also seems outdated. Can you please rebase?
Comment 258 Jonathan Druart 2015-01-12 08:09:42 UTC
Branch rebased against master.
Comment 259 Tomás Cohen Arazi 2015-01-13 16:32:11 UTC
Patches pushed to master.

Thanks Jonthan and everyone involved.
Hopefully no problems will arise, keep your eyes open to encoding issues!
Comment 260 Marcel de Rooy 2015-01-14 12:35:25 UTC
(In reply to Tomás Cohen Arazi from comment #259)
> Patches pushed to master.
> 
> Thanks Jonthan and everyone involved.
> Hopefully no problems will arise, keep your eyes open to encoding issues!

Great work, guys. This has been a lot of work!
So far I did not spot anything while testing with Latin and Chinese chars.
Except one small thing:

Frédéric Demians on the about page, timeline tab (reading the history txt file).
Comment 261 Jonathan Druart 2015-01-14 12:50:22 UTC
(In reply to M. de Rooy from comment #260)
> Frédéric Demians on the about page, timeline tab (reading the history txt
> file).

See bug 13579 for a fix.
Comment 262 Jonathan Druart 2015-01-14 15:42:05 UTC
I am going to reupload on this bug report the 31 patches pushed into master, for the history and/or backport them.
Comment 263 Jonathan Druart 2015-01-14 15:42:22 UTC
Created attachment 35223 [details] [review]
Bug 11944: Remove all utf8 filter from templates

This patch
- removes all html_entity usages in tt file which hide utf8 bugs
- removes all encode utf8 in tt plugins because we should get correctly
  marked data from DBIC and other sources directly (cf plugin EncodeUTF8
  used in renew.tt)
- adds some cleanup in C4::Templates::output: we now use perl utf8 file
  handler output so we don't need to decode tt variables manually.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 264 Jonathan Druart 2015-01-14 15:42:29 UTC
Created attachment 35224 [details] [review]
Bug 11944: replace use of utf8 with Encode

See the wiki page for the explanation.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 265 Jonathan Druart 2015-01-14 15:42:35 UTC
Created attachment 35225 [details] [review]
Bug 11944: Encoding UTF8 in C4::Templates

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 266 Jonathan Druart 2015-01-14 15:42:40 UTC
Created attachment 35226 [details] [review]
Bug 11944: Set ImplicitUnicode for YAML

Before this patch, string for yaml files used in preferences.pl were
broken.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 267 Jonathan Druart 2015-01-14 15:42:46 UTC
Created attachment 35227 [details] [review]
Bug 11944: Library name variable should be decoded

Test on preferences.pl and on some others pages when mysql is used to
store session.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 268 Jonathan Druart 2015-01-14 15:42:52 UTC
Created attachment 35228 [details] [review]
Bug 11944: replace uri_escape with uri_escape_utf8 everywhere

To reproduce, edit, index notice with utf-8 char and search for it

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 269 Jonathan Druart 2015-01-14 15:42:58 UTC
Created attachment 35229 [details] [review]
Bug 11944: Never use decode/encode in TT Plugins

The EncodeUTF8 TT plugin is useless, we should NEVER use encode in
templates.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 270 Jonathan Druart 2015-01-14 15:43:04 UTC
Created attachment 35230 [details] [review]
Bug 11944: Remove obsolete encoding routines

We don't need to decode manually which provide nice performance boost.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 271 Jonathan Druart 2015-01-14 15:43:12 UTC
Created attachment 35231 [details] [review]
Bug 11944: TT Plugins should not encode strings

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 272 Jonathan Druart 2015-01-14 15:43:19 UTC
Created attachment 35232 [details] [review]
Bug 11944: Some refresh don't indicate the charset to use

This patch replaces occurrences of META HTTP-EQUIV=Refresh with a
redirection.

Note:
 The list of modified files has been got with the following grep:
  git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js

Test plan:
1) admin/auth_tag_structure.pl
- Go on an authority MARC framework for an authority type
(admin/auth_tag_structure.pl?authtypecode=CO for instance)
- Click on the "Add a new tag" button
- Fill fields
- Save
- You should be redirected on the auth_tag_structure.pl page
- Try to modify an existing tag
- The redirection should be done

2) admin/auth_subfields_structure.pl
- From the previous page, click on a "subfields" link
- Edit/Add a new subfields
- Save
- You should be redirected to the subfield list
- Delete a subfield
- You should be redirected to the subfield list

3) admin/authorised_values.pl
- Add/Edit an authorised value
- Save
- You should be redirected to the authorised value list (with the
  right category selected)

4) admin/categorie.pl
- Add/Edit a patron category
- Save
- You should be redirected to the category list
- Delete a patron category
- You should be redirected to the category list

5) admin/koha2marclinks.pl
- Go on the Koha to MARC mapping configuration page
- Edit a field
- Map a field and click on "OK"
- You should be redirected to the list

6) admin/marc_subfields_structure.pl
- Go on the MARC frameworks page
- Click on "MARC structure" for one on the list
- Click on "Subfields" for one on the list
- Add/Edit a subfield
- Save
- You should be redirected to the subfield list

7) admin/systempreferences.pl
- Go on the system preferences admin page
- Click on the "Local use" tab
- Add/Edit a pref
- Save
- You shold be redirected to the Local use preference list

8) serials/subscription-detail.pl
- Go on a subscription detail page
- Delete the subscription (using Edit > Delete subscription)
- You should be redirected on the serials home page

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 273 Jonathan Druart 2015-01-14 15:43:26 UTC
Created attachment 35233 [details] [review]
Bug 11944: FIX Error on editing an authority

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 274 Jonathan Druart 2015-01-14 15:43:32 UTC
Created attachment 35234 [details] [review]
Bug 11944: Fix encoding issue on searching

The search patch should fix non-latin character searches.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 275 Jonathan Druart 2015-01-14 15:43:39 UTC
Created attachment 35235 [details] [review]
Bug 11944: Authentication

The password should be encoded before hashing.

Test plan:
- Before applying the patch, create a user with utf-8 in password
- apply patches
- try to log in
- change the password
- log out
- try to log in

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 276 Jonathan Druart 2015-01-14 15:43:45 UTC
Created attachment 35236 [details] [review]
Bug 11944: Fix fund names encoding on the parcel page

On the parcel page, the subtotal values are stored in an hashref. The
key was the fund name. The fund name being encoded in utf8, the hashref
was broken.
This patch replaces this logic creating a key based on an integer.
The budget name is now stored in a new "budget_name" key.

Note: It also fixes a small ergonomic issue (a td was missing in the
tfoot).

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 277 Jonathan Druart 2015-01-14 15:43:51 UTC
Created attachment 35237 [details] [review]
Bug 11944: Fix encoding issue on adding a subscription

If frequencies or numbering patterns contain non-latin characters, so
output was broken.

To reproduce:
- Create a frequency and a numbering pattern with non-latin characters
- Add a new subscription using these values
- The screen should not contain bad encoded characters.

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 278 Jonathan Druart 2015-01-14 15:43:58 UTC
Created attachment 35238 [details] [review]
Bug 11944: use CGI( -utf8 ) everywhere

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 279 Jonathan Druart 2015-01-14 15:44:05 UTC
Created attachment 35239 [details] [review]
Bug 11944: Fix encoding on sending emails

This patch fixes 2 places where mails were badly encoded:
1/ At the opac and the intranet, on sending baskets
2/ At the opac and the intranet, on sending shelf/list

Test plan:
Shelf/List:
- Create a list with non-latin characters in the name.
- Add some items containing non-latin characters in their content.
- Send the list by email

Basket/Cart:
- Add some items containing non-latin characters in their content to
  your cart..
- Send the cart by email

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 280 Jonathan Druart 2015-01-14 15:44:11 UTC
Created attachment 35240 [details] [review]
Bug 11944: Fix encoding issue for the "Did you mean" feature

Test plan:
1/ Go on admin/didyoumean.pl and enable the feature for the OPAC
2/ Generate and use the french templates
3/ Launch a search at the OPAC
4/ You should see correctly encoded message

Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 281 Jonathan Druart 2015-01-14 15:44:18 UTC
Created attachment 35241 [details] [review]
Bug 11944: Fix encoding issue on search history

Newly introduced by patches from bug 10807, the string was decoded
before to be stored in session.

To reproduce:
Enable history search
Go at the OPAC and log out
Search for something with utf-8
Go on your search history
The search description should be correctly encoded

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 282 Jonathan Druart 2015-01-14 15:44:24 UTC
Created attachment 35242 [details] [review]
Bug 11944: Remove bad FIXME in guided_report

Bug 11679 introduced a FIXME for this but it is wrong. The strings have
to be encoded before generating the ods file.

However, it is useless to decode column names.

Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 283 Jonathan Druart 2015-01-14 15:44:31 UTC
Created attachment 35243 [details] [review]
Bug 11944: Fix encoding issue in C4::ItemType

There is no need to encode strings coming from DB.

To reproduce:
go on admin/item_circulation_alerts.pl
The headers contain bad encoded characters.

Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 284 Jonathan Druart 2015-01-14 15:44:37 UTC
Created attachment 35244 [details] [review]
Bug 11944: (follow-up) Remove all utf8 filter from templates

This patch removes 2 new occurrences introduced by bug 11351 and bug
10493.

Signed-off-by: Magnus Enger <digitalutvikling@gmail.com>
Removes a nasty "Template process failed: plugin error - EncodeUTF8:
plugin not found at /home/magnus/scripts/kohaclone/C4/Templates.pm line 124"
from /cgi-bin/koha/admin/itemtypes.pl

Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 285 Jonathan Druart 2015-01-14 15:44:44 UTC
Created attachment 35245 [details] [review]
Revert "Bug 12279: Diacritics in system preference editor broken"

This reverts commit 2fa2480447a58f9d5d3f7834bcfdcf62a0a95df3.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>

Conflicts:
	admin/preferences.pl

Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>

http://bugs.koha-community.org/show_bug.cgi?id=11944

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 286 Jonathan Druart 2015-01-14 15:44:51 UTC
Created attachment 35246 [details] [review]
Bug 11944: remove url/uri filter from query_cgi

We don't need to pass query_cgi through url (or uri) filter since this
will turn query string (correctly escaped inside code) back into
two-byte escaped string which CGI( -utf8 ) won't turn back into proper utf-8

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 287 Jonathan Druart 2015-01-14 15:44:58 UTC
Created attachment 35247 [details] [review]
Bug 11944: decode uri_unescape data

This data is passed into form so it needs to be correctly marked as
utf-8 if we want form submit to work correctly

This change fixes sort issues which use form submit.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 288 Jonathan Druart 2015-01-14 15:45:06 UTC
Created attachment 35248 [details] [review]
Bug 11944: Convert DB tables to utf8_unicode_ci

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 289 Jonathan Druart 2015-01-14 15:45:14 UTC
Created attachment 35249 [details] [review]
Bug 11944: Fix suggestion at the OPAC for perl v5.10

Without this patch and using Perl v5.10, on adding a suggestion at the
OPAC, the script explodes with the following error:
"cannot decode string with wide characters".

With this patch, all is fine using Perl v5.10 and v5.18.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 290 Jonathan Druart 2015-01-14 15:45:22 UTC
Created attachment 35250 [details] [review]
Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci

Some lines didn't match the regex I used!

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 291 Jonathan Druart 2015-01-14 15:45:35 UTC
Created attachment 35251 [details] [review]
Bug 11944: revert unneeded IsStringUTF8ish behaviour change

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 292 Jonathan Druart 2015-01-14 15:45:42 UTC
Created attachment 35252 [details] [review]
Bug 11944: Fix problems introduced by bug 11515

The game continue...
Patch for bug 11515 introduced regression on this patch set.
To reproduce: search with utf8 characters at the opac
Test plan:
Verify that the issue described on bug 11515 is still fixed and that no
regression is found at the OPAC.

Note that I am pretty sure this patch is not enough.

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 293 Jonathan Druart 2015-01-14 15:45:50 UTC
Created attachment 35253 [details] [review]
Bug 11944: (RM followup) Missing dependency in C4::Context

Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Comment 294 Tomás Cohen Arazi 2015-05-19 15:17:57 UTC
*** Bug 10075 has been marked as a duplicate of this bug. ***
Comment 295 Tomás Cohen Arazi 2015-08-27 16:24:21 UTC
This bug is resolved already. The major refactoring it involved is already in 3.20 and further bugs / fixes should belong to their own bug reports.