I tried the solution from Bug 11290 In ...patron_lists/list.tt something like: [% p.borrowernumber.surname |html_entity %] It works for Skjöldóttur, but not for Беларуская I'm sorry to say I cannot reproduce the problem right now. I'm pretty sure this is related to some data being defaulted to ISO because of not explicitly marking as UTF8 (so is more related to latin+diacritics than non-latin) and hence breaking places where there is actual UTF-8 data. I had a similar problem with patron lists and was pointed to this: http://wiki.koha-community.org/wiki/Koha_on_ubuntu_-_tarball#Tweaking_my.cnf Making that change fixed it for me. Is that the correct solution, or is that a stopgap for a problem we need to address globally? (In reply to Owen Leonard from comment #3) > I had a similar problem with patron lists and was pointed to this: > > http://wiki.koha-community.org/wiki/Koha_on_ubuntu_-_tarball#Tweaking_my.cnf > > Making that change fixed it for me. Is that the correct solution, or is that > a stopgap for a problem we need to address globally? If the problem is environment settings-related (having MySQL set correctly is a pre-requisite for running Koha), this bug should be marked as invalid. I can volunteer to write something to check MySQL configuration sanity so we can (at least) ask users reporting this problems to visit about > system check to see if there's any problem. *** Bug 11942 has been marked as a duplicate of this bug. *** *** Bug 11949 has been marked as a duplicate of this bug. *** I followed the advice on http://wiki.koha-community.org/wiki/Koha_on_ubuntu_-_tarball#Tweaking_my.cnf and http://wiki.koha-community.org/wiki/Encoding_and_Character_Sets_in_Koha and added the following to: /etc/mysql/my.cnf ---------- [mysqld] init-connect = 'SET NAMES utf8' character-set-server=utf8 collation-server=utf8_general_ci character_set_client=utf8 ---------- Then I restarted mysql with: /etc/init.d/mysql restart As a result, I get the following in mysql: mysql> show variables like '%colla%'; +----------------------+-----------------+ | Variable_name | Value | +----------------------+-----------------+ | collation_connection | utf8_general_ci | | collation_database | utf8_general_ci | | collation_server | utf8_general_ci | +----------------------+-----------------+ mysql> show variables like '%char%'; +--------------------------+----------------------------+ | Variable_name | Value | +--------------------------+----------------------------+ | character_set_client | utf8 | | character_set_connection | utf8 | | character_set_database | utf8 | | character_set_filesystem | binary | | character_set_results | utf8 | | character_set_server | utf8 | | character_set_system | utf8 | | character_sets_dir | /usr/share/mysql/charsets/ | +--------------------------+----------------------------+ ...and the charecters now display fine. @ Tomás Cohen Arazi Having a MySQL configuration sanity check in about > system check would be great! Actually... those MySQL configuration changes aren't necessary in order to fix the problem. Something like this suffices: diff --git a/Koha/Database.pm b/Koha/Database.pm index 12758bf..ef1e2db 100644 --- a/Koha/Database.pm +++ b/Koha/Database.pm @@ -55,7 +55,7 @@ sub _new_schema { my $db_passwd = $context->config("pass"); my $schema = Koha::Schema->connect( "DBI:$db_driver:dbname=$db_name;host=$db_host;port=$db_port", - $db_user, $db_passwd ); + $db_user, $db_passwd, { mysql_enable_utf8 => 1 } ); return $schema; } This wouldn't be the final version of this patch, as we wouldn't want to pass msyql_enable_utf8 to a Pg database, and we would have to make sure that binary data (e.g., images stored in the database) don't get mangled, but I think something like this patch would be more reliable. Created attachment 26614 [details] [review] Bug 12001: regression test for using DBIC to store & fetch UTF8 strings This patch adds a regression test for verifying that a DBIx::Class schema object initialized by Koha database sets up the database connection to correct store and retrieve UTF8 values as Perl utf8 strings. To test: [1] Apply this patch. [2] Run prove -v t/db_dependent/Koha_Database.t [3] The test should fail. [4] Apply the main patch for this bug, then do step 2 again. [5] The test should pass. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Created attachment 26615 [details] [review] Bug 11906: ensure that Koha::Database uses UTF8 mode when connecting to databases This patch fixes an issue whereby the DBIx::Class schema objects were not connecting to the underlying database in UTF8 mode. This most visibility manifested as patron list pages not displaying diacritics correctly. To test: [1] Create a patron list, and make sure that it contains at least one patron whose name or patron category description contains a non-ASCII character. [2] View the list contents; the diacritics should appear mangled. [3] Apply the patch. [4] View the patron list again. This time, the diacritics should be displayed correctly. Note that Apache will also log "list.pl: Wide character in print ...", but this is the lesser of two evils. [5] Verify that prove -v t/db_dependent/Koha_Database.t passes. [6] (extra credit) Verify that t/db_dependent/Koha_Database.t passes when connect to a PostgreSQL database. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Created attachment 26616 [details] [review] Bug 11906: regression test for using DBIC to store & fetch UTF8 strings This patch adds a regression test for verifying that a DBIx::Class schema object initialized by Koha database sets up the database connection to correct store and retrieve UTF8 values as Perl utf8 strings. To test: [1] Apply this patch. [2] Run prove -v t/db_dependent/Koha_Database.t [3] The test should fail. [4] Apply the main patch for this bug, then do step 2 again. [5] The test should pass. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Created attachment 26617 [details] [review] Bug 11906: ensure that Koha::Database uses UTF8 mode when connecting to databases This patch fixes an issue whereby the DBIx::Class schema objects were not connecting to the underlying database in UTF8 mode. This most visibility manifested as patron list pages not displaying diacritics correctly. To test: [1] Create a patron list, and make sure that it contains at least one patron whose name or patron category description contains a non-ASCII character. [2] View the list contents; the diacritics should appear mangled. [3] Apply the patch. [4] View the patron list again. This time, the diacritics should be displayed correctly. Note that Apache will also log "list.pl: Wide character in print ...", but this is the lesser of two evils. [5] Verify that prove -v t/db_dependent/Koha_Database.t passes. [6] (extra credit) Verify that t/db_dependent/Koha_Database.t passes when connect to a PostgreSQL database. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Galen, We had this patch for bug 11944, so I completely agree with it! Actually it is not on bug 11944, it seems it disappears on rebasing/squashing patches :-/ I still have the patch on a local branch and the difference is: 1/ We added $schema->storage->on_connect_do('SET NAMES utf8'); in Koha::Database::_new_schema Which is stupid! From the mysql doc: "if you turn the flag on after connecting, you will need to issue the command SET NAMES utf8 to get the same effect." 2/ We did not propose unit tests, so ++ :) 3/ We removed the Koha::Template::Plugin::EncodeUTF8 uses in circ/renew.pl This plugin is now useless. I will submit a followup on QAing, feel free to ask open a new specific report for it. Marked as Passed QA. Created attachment 26622 [details] [review] Bug 11906: regression test for using DBIC to store & fetch UTF8 strings This patch adds a regression test for verifying that a DBIx::Class schema object initialized by Koha database sets up the database connection to correct store and retrieve UTF8 values as Perl utf8 strings. To test: [1] Apply this patch. [2] Run prove -v t/db_dependent/Koha_Database.t [3] The test should fail. [4] Apply the main patch for this bug, then do step 2 again. [5] The test should pass. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Created attachment 26623 [details] [review] Bug 11906: ensure that Koha::Database uses UTF8 mode when connecting to databases This patch fixes an issue whereby the DBIx::Class schema objects were not connecting to the underlying database in UTF8 mode. This most visibility manifested as patron list pages not displaying diacritics correctly. To test: [1] Create a patron list, and make sure that it contains at least one patron whose name or patron category description contains a non-ASCII character. [2] View the list contents; the diacritics should appear mangled. [3] Apply the patch. [4] View the patron list again. This time, the diacritics should be displayed correctly. Note that Apache will also log "list.pl: Wide character in print ...", but this is the lesser of two evils. [5] Verify that prove -v t/db_dependent/Koha_Database.t passes. [6] (extra credit) Verify that t/db_dependent/Koha_Database.t passes when connect to a PostgreSQL database. Signed-off-by: Galen Charlton <gmc@esilibrary.com> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz> Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Created attachment 26624 [details] [review] Bug 11906: The EncodeUTF8 TT Plugin is useless Now, the data from the DB, return by Koha::Database, have the utf-8 flag turn on. It becomes useless to encode them at the template level. The EncodeUTF8 plugin can be removed too. Test plan: 1/ Check out a book with a title containing non-latin characters. 2/ On circ/renew.pl, renew the item. 3/ In some case, you have a warning box, to override a limit. It should not contain broken strings. If you confirm, you get a "Item renewed" box, it should not contain broken strings. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> (In reply to Jonathan Druart from comment #13) > 3/ We removed the Koha::Template::Plugin::EncodeUTF8 uses in circ/renew.pl > This plugin is now useless. > I will submit a followup on QAing, feel free to ask open a new specific > report for it. Forget that, it does not work with translated template (the ENCODING flag is not set yet to 'UTF-8' at the template level). Great minds, and all that. Pushed to master. Pushed to 3.14.x, will be in 3.14.08 |
Created attachment 25916 [details] Bad display of utf-8 chars In patron lists patron names and other texts display ???? if they have umlauts and other utf-8 chars. Something like Беларуская Skjöldóttur dispays as ?????????? Skj�ld�ttur See screenshot attached.