This bug report will try to fix all UTF-8 problems once and forever! see http://wiki.koha-community.org/wiki/Handling_UTF8_in_development for more details about development guidelines for handling UTF-8 in Koha.
What would you guys think of pushing use feature 'unicode_strings' or use 5.012 and then fixing possible breakages?
Created attachment 26364 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually.
Created attachment 26365 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation.
Created attachment 26366 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates
Created attachment 26367 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken.
Created attachment 26368 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Created attachment 26369 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session.
Created attachment 26370 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with صة and search "صة"
Created attachment 26371 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates.
Created attachment 26372 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost.
(In reply to Tomás Cohen Arazi from comment #1) > What would you guys think of pushing > > use feature 'unicode_strings' > or > use 5.012 > > and then fixing possible breakages? Hi Tomás, we seen your comment too late, sorry :-/ I will have a look at unicode_strings ASAP!
I think this could be tested in order to catch bugs and have feedback quickly. Known bug: on importing an utf8 notice, the imported notice is broken.
URI de-/encoding needs some additional work - current version has serious uri encoding-related issues. To reproduce: try to search for some term (containing utf8 characters) in OPAC. Note: I tested OPAC searching with bootstrap theme (not sure if it may be relevant or not). E.g., searching for 'życie': /cgi-bin/koha/opac-search.pl?idx=&q=%C5%BCycie does return expected results, but links pointing to further/refined searches (facets, sorting, ..) on the result page are broken (pagination links are fine, though - probably because they are not being encoded at all). Offending links do have extra %25 's in unicode byte sequence: /cgi-bin/koha/opac-search.pl?idx=kw&q=%25C5%25BCycie&sort_by=relevance_asc&limit=itype:BK-KP
Created attachment 26407 [details] [review] Bug 11944: TT Plugins should not encode strings
Created attachment 26408 [details] [review] Bug 11944: Fix encoding issue on searching
Created attachment 26409 [details] [review] Bug 11944: Fix encoding issue on searching - latin characters The search is still broken for non-latin characters.
I insert a list of test for this bug: -- Install it on a marc21/english setup -- Import from files and from z39.50 greek/cyrillic records -- Index them in zebra, try searches and factes -- Catalogue update/display of record with greek/cyrillic chars -- Indexing update rsults -- libraries with greek/cyrillic names -- Try extra fields for patron with special chars [with system preference ExtendedPatronAttribute on 'YES'] -- Greek/cyrillic chars in ACQ budget/ordee/vendor -- Greek/cyrillic chars in letters
I test against master 3.15.00.26 The second patch doesn't apply: Applying: Bug 11944: replace use of utf8 with Encode Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging Koha/Database.pm CONFLICT (content): Merge conflict in Koha/Database.pm Failed to merge in the changes. This is ./Koha/Database.pm: 60 my $schema = Koha::Schema->connect( 61 "DBI:$db_driver:dbname=$db_name;host=$db_host;port=$db_port", 62 <<<<<<< HEAD 63 $db_user, $db_passwd, $db_opts ); 64 ======= 65 $db_user, $db_passwd ); 66 67 >>>>>>> Bug 11944: replace use of utf8 with Encode 68 return $schema; 69 }
Created attachment 26737 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually.
Created attachment 26738 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation.
Created attachment 26739 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates
Created attachment 26740 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken.
Created attachment 26741 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Created attachment 26742 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session.
Created attachment 26743 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session.
Created attachment 26744 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with صة and search "صة"
Created attachment 26745 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates.
Created attachment 26746 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost.
Created attachment 26747 [details] [review] Bug 11944: TT Plugins should not encode strings
Created attachment 26748 [details] [review] Bug 11944: Fix encoding issue on searching - latin characters The search is still broken for non-latin characters.
I apply against 3.15.00.026. In "Authorized values" of Administration module, I add a "New category", whose name is (newcat-mu): newcat-μ 1) ERROR: Soon after I save it, 3 greek chars are shown badly here: "New authorized values for newcat-μ:" "Authorized values for category newcat-μ:" "There are no authorized values defined for newcat-μ" The current URL is bad too: http://192.168.40.128:8080/cgi-bin/koha/admin/authorised_values.pl?searchfield=newcat-μ&offset=0 2) OK: The new category has been added to the list "Show category". Its name is shown OK. Now I select the new category "newcat-μ" inside this list, and I select "Show category". Koha answers with a page where all the 5 occurrences of newcat-μ are shown OK. So I pass to Failed QA status.
(In reply to Paola Rossi from comment #31) > In "Authorized values" of Administration module, I add a "New category", > whose name is (newcat-mu): > newcat-μ > > 1) ERROR: > Soon after I save it, 3 greek chars are shown badly here: > "New authorized values for newcat-μ:" > "Authorized values for category newcat-μ:" > "There are no authorized values defined for newcat-μ" Hum, weird, I did not reproduce on my local instance. Same on sandbox 17: http://pro.test17.biblibre.com/cgi-bin/koha/admin/authorised_values.pl Templates fr-FR and it-IT generated.
Dear Mr. Jonathan, two images are available at: 1)http://koha.wdfiles.com/local--files/immaginibugs/newcat-mu.png 2)http://koha.wdfiles.com/local--files/immaginibugs/newcat2-mu.png The first image shows the error, while the second one shows correct chars. The second one shows the already added category. The first one shows the moment soon after the adding. Your test at comment 32 references to the second moment, where no error is shown. When I cancel the category too, I see wrong chars. In the Sandbox 17, I recreated the error, see link: 3) http://www.koha.it/local--files/immaginibugs/newcat3-mu.png I use Firefox v.28.0 on Windows 7 environment.
(In reply to Paola Rossi from comment #33) > Your test at comment 32 references to the second moment, where no error is > shown. I create the new category after applying patches? > In the Sandbox 17, I recreated the error, see link: > 3) http://www.koha.it/local--files/immaginibugs/newcat3-mu.png Could you please describe, step by step, how do you arrive on this page? From http://pro.test17.biblibre.com/cgi-bin/koha/admin/authorised_values.pl I select "Italiano" in languages, I select "newcat-μ" in the dropdown list. And I see the table with my authorized value, correctly displayed. The only problem I see is that the dropdown list does not preselect the good value. What is the character encoding of your browser? (Edition > Character encoding)
IOO the error depended on the 3 lines of ./admin/authorised_values.pl: print "Content-Type: text/html ........... (165,195,227) because of the lack of charset="utf-8". Now the 165-th line appears so: 165 print "Content-Type: text/html; charset=\"utf-8\"...... We add charset="utf-8" in these 3 lines in our installation, and now everything's OK. Now the current category is correctly shown in the dropdown list, even if its name contains some UTF-8 chars. So now the dropdown list preselects the good value. The error appeared only with Firefox 28.0; it didn't appear with Explorer v.10 nor with Chrome v.33. With Zeno some lines like those are similar in others cgi. See f.i.: <http://paste.koha-community.org/204>
(In reply to Paola Rossi from comment #35) > IOO the error depended on the 3 lines of ./admin/authorised_values.pl: > > print "Content-Type: text/html ........... > > (165,195,227) > > because of the lack of charset="utf-8". Great! I am going to provide a followup for that!
Created attachment 26794 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page
Regarding problem described in comment #13 (this issue is not yet fixed in current version): it seems to be caused by / related to '|url' TT filter usage. In OPAC prog theme facets links are not affected; difference being that includes/opac-facets.inc in prog is using (a lot of) |html instead of |url. Problems with sorting criteria change/link not working are occurring in OPAC bootstrap & prog themes and in the staff searches too. Probably the same underlying cause.
(In reply to Jonathan Druart from comment #37) > Created attachment 26794 [details] [review] [review] > Bug 11944: Some refresh don't indicate the charset to use > > This patch replaces occurrences of META HTTP-EQUIV=Refresh with a > redirection. > > Note: > The list of modified files has been got with the following grep: > git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator > | grep -v \.js > > Test plan: [...] All the 8 tests are OK.
So here and there we uri-escape some strings twice (mostly various search query parts, plus some more): - directly, by uri_escape_utf8() followed by '|url' or '|uri' TT filter, - indirectly, by using already uri-escaped string in <input value="..."> which will trigger 2nd uri-escape in browser The other way around, they are uri-unescaped twice too: first in CGI.pm and 2nd time directly by using uri_unescape in Search.pm etc. (including some places when they rather shouldn't; funny things might happen when searching for something with '%' in it..). But what's more important, delayed 2nd uri_unescape() would prevent those strings from being promoted to 'wide character' despite CGI qw(-utf8). This terribly dirty fix seems to resolve both problems mentioned in comment #13 (in OPAC search; staff search may need another one): diff --git a/opac/opac-search.pl b/opac/opac-search.pl index 7a35085..db0545e 100755 --- a/opac/opac-search.pl +++ b/opac/opac-search.pl @@ -387,7 +387,8 @@ if ($indexes[0] && !$indexes[1]) { } # an operand can be a single term, a phrase, or a complete ccl query my @operands = $cgi->param('q'); -@operands = map { uri_unescape($_) } @operands; +## Not a proper fix, just proof-of-concept!! +@operands = map { /%[a-f\d]{2}/i? Encode::decode('UTF-8', uri_unescape($_)): $_; } @operands;
New-ing or updating a patron, a password like this one: 実provo causes the error: Software error: input must contain only octets at /usr/lib/perl5/Crypt/Eksblowfish/Bcrypt.pm line 84. For help, please send mail to the webmaster (webmaster@linux-test), giving this error message and the time and date of the error. Against master, the error doesn't appear. NB. From Koha devel, from M.de.Rooy@rijksmuseum.nl: Quoting from a previous try to get UTF8- moving, report 9951: [[ A nice one to start with: opac-password.pl: The password may contain diacritics. It should be decoded. But this needs a dive into C4/Auth.pm also (and probably in staff too). And we still have the problem with old incorrect hashes in the table. If you would have an old password with diacritics, you would still like to login. This is not just adding a flag to CGI, but a report on its own. So I leave that for now; these hashes still work ]] How would you suggest to resolve the old passwords given that Koha and used perl libs would encode/decode properly ? No reason not to support it, but it needs some more thought.. ----------------------------------------------------------------- So I pass the status to "Failed QA".
(In reply to Paola Rossi from comment #41) > input must contain only octets at /usr/lib/perl5/Crypt/Eksblowfish/Bcrypt.pm > line 84. Hm, we can't realy expect Bcrypt to handle perl wide-characters reliably, those hashing algorithms were designed to deal with octets/bytes only.. In particular, if we want all hashes allready stored in the database still to work, the only reasonable solution IMO will be to hash passwords in utf-8 encoded form - and preferably to keep them in as much unaltered form as possible before hashing. Right now we get password params already decoded thanks to CGI qw( -utf8); I guess we can [re]encode them before hashing, but there is no guarantee result string will be allways the same after utf-8 encode(decode()). BTW, is there some way to fetch on demand a specific parameter from CGI.pm in non-decoded form, i.e. bypassing en effect of qw( -uft8)?
Created attachment 26857 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually.
Created attachment 26858 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation.
Created attachment 26859 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates
Created attachment 26860 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken.
Created attachment 26861 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Created attachment 26862 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session.
Created attachment 26863 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with صة and search "صة"
Created attachment 26864 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates.
Created attachment 26865 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost.
Created attachment 26866 [details] [review] Bug 11944: TT Plugins should not encode strings
Created attachment 26867 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page
Created attachment 26868 [details] [review] Bug 11944: FIX Error on editing an authority
Created attachment 26869 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches.
Created attachment 26870 [details] [review] Bug 11944: Facets and resort This patch should fix facets and resort encoding issues. Note: This patch will certainly break the search history
Created attachment 26871 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with a password "μμμμ" - apply patches - try to log in - change the password - log out - try to log in
(In reply to Paola Rossi from comment #41) > New-ing or updating a patron, a password like this one: > > 実provo > > causes the error: This should be fixed by patch "Bug 11944: Authentication" (In reply to Jacek Ablewicz from comment #40) It seems I managed to fix that in patch "Bug 11944: Facets and resort". But I'm not really sure... Last patch set fix some problems in previous patches: Encode::decode does not work on MARC::Record (of course!) Encode::decode returns the decoded value (and does not modify the variable passed in parameter). I don't reproduce the "importing" error any longer, it seems fixed. Please add known bugs on the wiki page: http://wiki.koha-community.org/wiki/Handling_UTF8_in_development#Known_bugs
Today I've installed master 3.15.00.027 I try to apply the patches. At the 6-th patch there a conflict, perhaps due to the rebasing: Applying: Bug 11944: use CGI( -utf8 ) everywhere Using index info to reconstruct a base tree... <stdin>:2009: trailing whitespace. use CGI qw ( -utf8 ); warning: 1 line adds whitespace errors. Falling back to patching base and 3-way merge... Auto-merging tools/koha-news.pl CONFLICT (content): Merge conflict in tools/koha-news.pl Auto-merging opac/opac-main.pl CONFLICT (content): Merge conflict in opac/opac-main.pl Auto-merging mainpage.pl CONFLICT (content): Merge conflict in mainpage.pl Failed to merge in the changes. Patch failed at 0001 Bug 11944: use CGI( -utf8 ) everywhere ------------------ So I pass the patch to "Patch doesn't apply" status.
Created attachment 26895 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Last patch resolves conflicts with bug 7567.
In ACQ, I create a fund whose name is "fund1 μμμ" for a budget. I order a biblio, and I receive the order. In the "Receipt summary", in the table of the "Already received", I see: 1.A) in the 8-th column "Fund", the name of the fund with the right chars; 1.B) in the 7-th column "Quantity", at the "Subtotal for", the name of the fund with bad chars: "Subtotal for fund1 μμμ". ----------------------------------------------------------------- So I pass the status to "Failed QA".
Created attachment 26921 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot).
I add a new UniMARC greek authority, i.e.: .... 200$aΜαρωνίτης, 200$bΔ. Ν. .... The db contains the right chars; from export of the db: ...<subfield code=\"a\">Μαρωνίτης,</subfield>\n <subfield code=\"b\">Δ. Ν.</subfield>.... When I see the new authority on: cgi-bin/koha/authorities/detail.pl?authid=1 koha shows tag 200 with the wrong chars, such as: a Entry Element ÎαÏÏνίÏηÏ, b Part of Name Other t Î. Î. Nel source: .... <input type="text" id="tag_200_subfield_a_818751_534076" name="tag_200_subfield_a_818751_534076" value="ÎαÏÏνίÏηÏ," tabindex="1" size="67" maxlength="9999" class="input_marceditor" /> ....etc. These wrong chars are shown in FF, Chrome and Explorer. ----------------------------------------------------------- So I pass the status to "Failed QA".
The problem in comment 64 about authorities in koha UniMARC doesn't occur in koha MARC21. Perhaps the wrong detail occured because of the bad sql file for UniMARC in English set up. So the status is "Needs Signoff" again.
In Serials, I create a new numbering pattern whose name is: new実 numbering pattern In this pattern I set this Label: 実label When I select this new pattern for a subscription, the label shown has bad chars: å®label I go on and select "Show advanced pattern". The red "Pattern name" shown has bad chars: newå® numbering pattern ----------------------------------------------------------- So I pass the status to "Failed QA".
Created attachment 27021 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters.
I apply the last "Fix encoding issue" patch. No bad chars are seen, But, if I use f.i. the "frequency" 1/day, in the "Advanced prediction pattern" shown table "null" is shown in some cells. In sandbox 17 no "null" is shown in these cells. IMO this could be a bug, and so I pass the patch to "Failed QA" status.
Created attachment 27080 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters.
I create a list whose name is: 学協会. μμ I add some biblios to the list [the titles of the added biblios contain greek and japanese chars]. I try to "Send list" to my email. Either I give over-255 chars in the "Comment" of the mail, or I don't give any over-255 char in the "Comment" of the mail, I see this abnormal termination: Software error: Wide character in subroutine entry at /var/root-koha/bug-11944/virtualshelves/sendshelf.pl line 134. I use the 17 sandbox. I create a list whose name is: 学協会. μμ I add some biblios to the list [the titles of the added biblios contain greek and japanese chars]. Either I give over-255 chars in the "Comment" of the mail, or I don't give any over-255 char in the "Comment" of the mail, the answer of koha is right: Messaggio inviato La lista è stata mandata a: paola.rossi@cineca.it Chiudi la finestra So I receive the expected mail. The received file is OK with right chars, while the subject of the mail is the string "no subject". So I pass the patch to "Failed QA" status. NB. In the messagge: La lista è stata mandata a: ... [i.e. the list has been sent to ...] the name of the list is not shown, whilest I might select among a lot of lists of mine. So koha gives no feedback about which list has been sent, and I can't see the list name on this page.
Created attachment 27094 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email
1) About sending (=Envoyer) Basket/Cart: Basket/Cart is sent OK: - either on intranet or on OPAC, - with over-255 chars either in the content of the mail (i.e. Commentaire) or in the content of the attached file (i.e. biblios titles), - by English, by French and by Italian But: the French message for cart shows bad chars: Bonjour, 113 vous a envoy� un panier de r�f�rences extraites de notre catalogue. Notez que le fichier attach� est un fichier MARC, qui peut �tre import� dans un logiciel de gestion de bibliographie tel que EndNote, Reference Manager ou ProCite. the Italian message for cart shows bad chars: Ciao, 113 ti ha inviato un carrello dal nostro catalogo online. Il file in allegato � fatto di record MARC che possono essere importati in un software bibliografico come EndNote, Reference Manager, ProCite, ecc. ------------------------------------------------------------------------------ 2) About sending (=Envoyer) Shelf/List: on OPAC, list is sent OK: - with over-255 chars either in the content of the mail (i.e. Commentaire) or in the content of the attached file (i.e. biblios titles), - by English, by French and by Italian. On intranet: -- French doesn't send any list [with or without over-255 chars (in biblios titles and in the list name)]; -- Italian doesn't send any list [with or without over-255 chars (in biblios titles and in the list name)]; [i.e. Italian same as French] -- English sends the list [with or without over-255 chars (in biblios titles and in the list name)]; the attached file is ok, while the content of the mail shows bad chars over-127 (biblionumber 1 in sand box 17: La Recherche å¦åä¼. μμ by Published by: Stépahanie KHEMIS, 57 -------------------------------------------------------------- The attached files are all OK. -------------------------------------------------------------- So I pass the patch to "Failed QA" status.
(In reply to Paola Rossi from comment #72) Arg, it works for me. Which email client do you use?
I and Paola we use Thunderbird 24.4.0
Created attachment 27158 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Created attachment 27159 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email
This patch set now depends on bug 11248.
(In reply to Zeno Tajoli from comment #74) > I and Paola we use Thunderbird 24.4.0 I reproduced using icedove. The "header" was not encoded. It should work now.
Created attachment 27161 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see "Voulez-vous dire : Chercher également les termes spécifiques Chercher également les termes génériques Chercher également les termes reliés" Before this patch: "Voulez-vous dire : Chercher �galement les termes sp�cifiques Chercher �galement les termes g�n�riques Chercher �galement les termes reli�s"
I apply against master 3.15.00.030 and at the 16-th patch: Applying: Bug 11944: Fix encoding issue on adding a subscription fatal: sha1 information is lacking or useless (serials/subscription-frequency.pl). Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 11944: Fix encoding issue on adding a subscription ----------------------------------------------------------------------------- So I pass the patch to "Patch doesn't apply" status.
Created attachment 27193 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters.
Created attachment 27194 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Created attachment 27195 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email
Created attachment 27196 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see "Voulez-vous dire : Chercher également les termes spécifiques Chercher également les termes génériques Chercher également les termes reliés" Before this patch: "Voulez-vous dire : Chercher �galement les termes sp�cifiques Chercher �galement les termes g�n�riques Chercher �galement les termes reli�s"
I apply against master 3.15.00.030 and at the 17-th patch: Applying: Bug 11944: use CGI( -utf8 ) everywhere fatal: sha1 information is lacking or useless (opac/opac-sendshelf.pl). Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 11944: use CGI( -utf8 ) everywhere ----------------------------------------------------------------------------- So I pass the patch to "Patch doesn't apply" status.
Created attachment 27198 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually.
Created attachment 27199 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation.
Created attachment 27200 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates
Created attachment 27201 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken.
Created attachment 27202 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session.
Created attachment 27203 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with صة and search "صة"
Created attachment 27204 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates.
Created attachment 27205 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost.
Created attachment 27206 [details] [review] Bug 11944: TT Plugins should not encode strings
Created attachment 27207 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page
Created attachment 27208 [details] [review] Bug 11944: FIX Error on editing an authority
Created attachment 27209 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches.
Created attachment 27210 [details] [review] Bug 11944: Facets and resort This patch should fix facets and resort encoding issues. Note: This patch will certainly break the search history
Created attachment 27211 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with a password "μμμμ" - apply patches - try to log in - change the password - log out - try to log in
Created attachment 27212 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot).
Created attachment 27213 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters.
Created attachment 27214 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere
Created attachment 27215 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email
Created attachment 27216 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see "Voulez-vous dire : Chercher également les termes spécifiques Chercher également les termes génériques Chercher également les termes reliés" Before this patch: "Voulez-vous dire : Chercher �galement les termes sp�cifiques Chercher �galement les termes g�n�riques Chercher �galement les termes reli�s"
I resubmit all 19 patches.
I'm sorry, but I see the problem again with the new patches. The problem is still on the 17-th patch, about the same pl: Applying: Bug 11944: use CGI( -utf8 ) everywhere fatal: sha1 information is lacking or useless (opac/opac-sendshelf.pl). Repository lacks necessary blobs to fall back on 3-way merge. Cannot fall back to three-way merge. Patch failed at 0001 Bug 11944: use CGI( -utf8 ) everywhere ----------------------------------------------------------------------------- This is the opac/opac-sendshelf.pl from 20-th line on: use strict; use warnings; use CGI; use Encode; use Carp; .... ----------------------------------------------------------------------------- whilest these are the lines searched by the 17-th patch: -use CGI; -use Encode qw(decode encode); ----------------------------------------------------------------------------- So I pass the patch to "Patch doesn't apply" status.
(In reply to Paola Rossi from comment #106) It should work. Did you apply patches from bug 11248 before?
(In reply to Jonathan Druart from comment #107) > (In reply to Paola Rossi from comment #106) > > It should work. Did you apply patches from bug 11248 before? That was the error! Now I've started applying the 3 patches of 11248, and so I can apply all the 22 patches. So I pass the patch to "Needs Signoff" status. [Until May the 5-th I'll rarely be on bugzilla.]
I test against master 3.15.00.036. [Preference UseICU set to "Using"] I've tested ACQ, Serials and the Advanced Search module. IMO the patch is OK about these 3 modules. So I pass the patch to "Signed Off" status. ----------------------------------------------------------------------------- Some doc about advanced search test. I create some [fictious] authorities (personal name' author, topical term' subject) with Japanese (such as 本学会誌), Greek (Σῖγμασς), Arabic (مكن إجراء البحث من), and some other (ČčĕīōŠ) chars. Then I fill the 700$a and 650$a of some biblios with them. Titles of the biblios have those chars too. Some fields are like the following ones (1 alphabet's chars only): يمكن إجراء البحث من Σῖγμα, σῖγςμας 本, 学会誌 whilest other fields are mixture (!) of some alphabets' chars, such as: Čajko author Ibn Gĕbīrōl, Šĕlōmōh ben Yĕhudāh يمكن إجراء البحث من Alcock 実 μσῖ, Ian Č topic mu μμ Gĕbīrōl, Šĕlōmōh ben Yĕhudāh, Petr Ilʹič I think the mixture fields are perhaps not real cases: anyway I've checked the koha's behaviour on these hypothetical cases too. Searching is OK (keyword, keyword as phrase, Author, Author as phrase, Personal name, Personal name as phrase, Subject, Subject as phrase). Facets are created well. The link from a facet is OK (the answer is correct), with the exception of NB1. NB1. The mixed fields with 2 [or more] adjacent Japanese chars are well listed in facets, but these facets have bad links, selecting erroneuosly nothing; i.e. these kind of facets find nothing [because of the two adjacent Japanese chars inside]: No results match your search for 'kw,wrdl: ...' with limit(s): 'au:Σῖγμαprovo実会 mu μμμ, σῖγςμας' NB2. If I search for 'Č' in : advanced search, f.i. by "Keyword" field, [in ADVANCED SEARCH] search authorities by "Contains" operator, [in AUTHORITIES] search authorities by Tag editor, f.i. by "Main Entry contains" [in CATALOG] the answer is the same as the answer against searching for "c". Perhaps this is the expected behaviour. -----------------------------------------------------------------------------
Created attachment 27585 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27586 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27587 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27588 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27589 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27590 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with صة and search "صة" Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27591 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27592 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27593 [details] [review] Bug 11944: TT Plugins should not encode strings Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27594 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27595 [details] [review] Bug 11944: FIX Error on editing an authority Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27596 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27597 [details] [review] Bug 11944: Facets and resort This patch should fix facets and resort encoding issues. Note: This patch will certainly break the search history Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27598 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with a password "μμμμ" - apply patches - try to log in - change the password - log out - try to log in Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27599 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot). Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27600 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters. Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27601 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27602 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
Created attachment 27603 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see "Voulez-vous dire : Chercher également les termes spécifiques Chercher également les termes génériques Chercher également les termes reliés" Before this patch: "Voulez-vous dire : Chercher �galement les termes sp�cifiques Chercher �galement les termes g�n�riques Chercher �galement les termes reli�s" Signed-off-by: Paola Rossi <paola.rossi@cineca.it>
(In reply to Paola Rossi from comment #109) > So I pass the patch to "Signed Off" status. Thanks Paola! I added your signoff to the patch set. I switch the status back to Needs Signoff for a second signoff.
Marking as candidate for 3.16.0.
Created attachment 27880 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> My comments on last patch.
Created attachment 27881 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27882 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27883 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27884 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
I'm in the process of sign, but have problems to upload some patches with git-bz. I'm out for a moment, will finish soon
Created attachment 27885 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with XXX and search "XXX" Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> XXX = removed char, because of git-bz
Created attachment 27886 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27887 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27888 [details] [review] Bug 11944: TT Plugins should not encode strings Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27889 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27890 [details] [review] Bug 11944: FIX Error on editing an authority Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27891 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27892 [details] [review] Bug 11944: Facets and resort This patch should fix facets and resort encoding issues. Note: This patch will certainly break the search history Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27893 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with a password "XXXX" - apply patches - try to log in - change the password - log out - try to log in Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> replaced string with XXXX, sorry
Created attachment 27894 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot). Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27895 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27896 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27897 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 27898 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see [ Trimmed text, look original patch :( ] Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Comments: No koha-qa erros, save for a FAIL on first patch for enabling utf8, and third to last patch, because it take ages and not finish :) But there are simple changes, I think there must no be a problem As with many enconding problems, some I can not reproduce, but some yes, and that are fixed For example: 1) password (no problem before or after) 2) fund (no problem before or after) 3) frecuency (no problem before or after) 4) Numbering pattern (no problem before or after) 5) List (shelfs), content, mail (bad before, fixed) 6) Cart content, mail (bad before, fixed) 7) didyoumean (no problems before or after) Works for me.
Sorry, I need to rewrite some text on commit messages. But is finished
On reflection -- I'm not going to bypass QA on this one, given its size. Remove the 3.16 candidate keyword.
Do you think it is still possible to insert it ? In fact it doesn't change template strings so it is not blocked by string freeze.
I think it would be wise not to push it into master before the release. But it would be great to have the patches into 3.17 asap, in order to catch remaining bugs by the developer's team.
Created attachment 28046 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28047 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28048 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28049 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28050 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28051 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with صة and search "صة" Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28052 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28053 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28054 [details] [review] Bug 11944: TT Plugins should not encode strings Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28055 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28056 [details] [review] Bug 11944: FIX Error on editing an authority Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28057 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28058 [details] [review] Bug 11944: Facets and resort This patch should fix facets and resort encoding issues. Note: This patch will certainly break the search history Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28059 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with a password "μμμμ" - apply patches - try to log in - change the password - log out - try to log in Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28060 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot). Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28061 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28062 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28063 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28064 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see "Voulez-vous dire : Chercher également les termes spécifiques Chercher également les termes génériques Chercher également les termes reliés" Before this patch: "Voulez-vous dire : Chercher �galement les termes sp�cifiques Chercher �galement les termes g�n�riques Chercher �galement les termes reli�s" Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
I rebased my local branch against master, added the Bernardo'SO to the patches and reuploaded them. Original commit messages are present.
Created attachment 28065 [details] [review] Bug 11944: Fix encoding issue on search history Newly introduced by patches from bug 10807, the string was decoded before to be stored in session. To reproduce: Enable history search Go at the OPAC and log out Search for صة Go on your search history The search description should be correctly encoded Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 28066 [details] [review] Bug 11944: Remove bad FIXME in guided_report Bug 11679 introduced a FIXME for this but it is wrong. The strings have to be encoded before generating the ods file. However, it is useless to decode column names. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
(In reply to Jonathan Druart from comment #176) > However, it is useless to decode column names. I am not sure about that..
Great to see this thorough work on utf8! We have done a lot of manual corrections on .pref files previously. However, applying these patches on a clean master and running translate install nb-NO now leads to EVERY bit of koha becoming double encoded... example: Advanced search | Authority search | Tag cloud becomes: Avansert søk | Autoritetssøk | Taggsky is there something I'm missing? I've tried swedish as well, with same result.
(In reply to Benjamin Rokseth from comment #178) > example: > Advanced search | Authority search | Tag cloud > becomes: > Avansert søk | Autoritetssøk | Taggsky > > > is there something I'm missing? I've tried swedish as well, with same result. I just tried and it seems to work: Avansert søk Course reserves Autoritetssøk Taggsky Emnesky Mest populære At the OPAC, on the main page.
I just found a bug with the search, it's completely broken.
Created attachment 28097 [details] [review] Bug 11944: Facets and resort
(In reply to Jonathan Druart from comment #181) > Created attachment 28097 [details] [review] [review] > Bug 11944: Facets and resort I need more testing here. This patch does not fix everything but I didn't manage to list what is broken or not. I need some expert too: what is the best way to handle the uri_escape/unescape and the uri, url, html TT filters? There are a lot of all of that in include files, results.tt, search.pl and Seach.pm...
(In reply to Jonathan Druart from comment #182) > (In reply to Jonathan Druart from comment #181) > I need some expert too: what is the best way to handle the > uri_escape/unescape and the uri, url, html TT filters? > There are a lot of all of that in include files, results.tt, search.pl and > Seach.pm... Looking at unescape: uri_escape_utf8 encodes data before escaping, so expects perl internal format (what you should normally have..) uri_escape and uri_unescape operate on byte level; so you should normally expect to decode their output in your program. Also note that CGI normally handles escaping/unescaping for params. So it seems that some unescaping in catalogue/search.pl and opac-search.pl is useless. The unescape in members/paycollect.pl is not decoded. Could be a problem. In cataloguing/additem a cookie is unescaped and thawed. Should be checked. This one is confusing (opac-search): my @query_inputs = _input_cgi_parse($query_cgi); $template->param ( QUERY_INPUTS => \@query_inputs ); $query_cgi comes from buildQuery and contains uri_escape_utf8ed string (so: encoded data) _input_cgi_parse runs a part through uri_unescape, so still encoded. we pass it to the template in encoded form: Does it get mixed up with decoded data here? C4/Search/History: decode_json( uri_unescape( $session->param('search_history') ) ) $session is a CGI Session object: How does it handle utf8? uri_unescape operates on byte level decode_json expects an encoded string Should be checked. Not sure if the output is encoded or decoded in perl.
(In reply to Benjamin Rokseth from comment #178) > However, applying these patches on a clean master and running > translate install nb-NO > now leads to EVERY bit of koha becoming double encoded... I worked with Benjamin last week to try and recreate this, but we did not find a solution. We were both on Ubuntu 14.04, with the same setup and the same versions of important modules. Benjamin's templates were broken and mine were not. The only difference was that I had cloned the code from git.koha-community.org and Benjamin had cloned from some mirror (due to firewall restrictions).
Comment on attachment 28062 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Review of attachment 28062 [details] [review]: ----------------------------------------------------------------- ::: members/guarantor_search.pl @@ +24,4 @@ > #use warnings; FIXME - Bug 2505 > use C4::Auth; > use C4::Output; > +use CGI qw ( -utf8 ); We have a merge conflict against master now. :(
Comment on attachment 28063 [details] [review] Bug 11944: Fix encoding on sending emails Review of attachment 28063 [details] [review]: ----------------------------------------------------------------- This patch has merge conflicts. :(
Created attachment 28245 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28246 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28247 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see "Voulez-vous dire : Chercher également les termes spécifiques Chercher également les termes génériques Chercher également les termes reliés" Before this patch: "Voulez-vous dire : Chercher �galement les termes sp�cifiques Chercher �galement les termes g�n�riques Chercher �galement les termes reli�s" Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Created attachment 28248 [details] [review] Bug 11944: Fix encoding issue on search history Newly introduced by patches from bug 10807, the string was decoded before to be stored in session. To reproduce: Enable history search Go at the OPAC and log out Search for صة Go on your search history The search description should be correctly encoded Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 28249 [details] [review] Bug 11944: Remove bad FIXME in guided_report Bug 11679 introduced a FIXME for this but it is wrong. The strings have to be encoded before generating the ods file. However, it is useless to decode column names. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 28250 [details] [review] Bug 11944: Facets and resort
(In reply to M. Tompsett from comment #186) > Comment on attachment 28063 [details] [review] [review] > This patch has merge conflicts. :( Last 6 patches rebased.
Overall, looking pretty good with Norwegian characters now, thank you for your efforts! So far I only found two issues: material-types are double encoded in the item circulation alerts table: /cgi-bin/koha/admin/item_circulation_alerts.pl which made me want to check material types at /cgi-bin/koha/admin/itemtypes.pl but it fails with Template process failed: plugin error - EncodeUTF8: plugin not found at /usr/local/src/kohaclone/C4/Templates.pm line 124.
Created attachment 28251 [details] [review] Bug 11944: Fix encoding issue in C4::ItemType There is no need to encode strings coming from DB. To reproduce: go on admin/item_circulation_alerts.pl The headers contain bad encoded characters.
(In reply to Petter Goksøyr Åsen from comment #194) > So far I only found two issues: > material-types are double encoded in the item circulation alerts table: > > /cgi-bin/koha/admin/item_circulation_alerts.pl > > which made me want to check material types at > > /cgi-bin/koha/admin/itemtypes.pl > > but it fails with > > Template process failed: plugin error - EncodeUTF8: plugin not found at > /usr/local/src/kohaclone/C4/Templates.pm line 124. I can confirm both of these problems. It looks like the EncodeUTF8 plugin is removed in attachment 28052 [details] [review], but it is still used in a number of places, after the patches on this bug have been applied: $ grep -rl --exclude=*.po "EncodeUTF8" * koha-tmpl/intranet-tmpl/prog/en/modules/admin/itemtypes.tt koha-tmpl/intranet-tmpl/prog/en/modules/circ/renew.tt koha-tmpl/intranet-tmpl/prog/nb-NO/modules/admin/itemtypes.tt koha-tmpl/intranet-tmpl/prog/nb-NO/modules/circ/renew.tt
Created attachment 28252 [details] [review] Bug 11944: (follow-up) Remove all utf8 filter from templates This patch removes 2 new occurrences introduced by bug 11351 and bug 10493.
Created attachment 28253 [details] [review] [SIGNED-OFF] Bug 11944: (follow-up) Remove all utf8 filter from templates This patch removes 2 new occurrences introduced by bug 11351 and bug 10493. Signed-off-by: Magnus Enger <digitalutvikling@gmail.com> Removes a nasty "Template process failed: plugin error - EncodeUTF8: plugin not found at /home/magnus/scripts/kohaclone/C4/Templates.pm line 124" from /cgi-bin/koha/admin/itemtypes.pl
Comment on attachment 28252 [details] [review] Bug 11944: (follow-up) Remove all utf8 filter from templates Uploaded a signed off version of this
Today I try to apply against master 3.17.00.007 The first patch doesn't apply: Applying: Bug 11944: Remove all utf8 filter from templates Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging koha-tmpl/opac-tmpl/prog/en/modules/opac-account.tt CONFLICT (content): Merge conflict in koha-tmpl/opac-tmpl/prog/en/modules/opac-account.tt Auto-merging Koha/Template/Plugin/Branches.pm Failed to merge in the changes. Patch failed at 0001 Bug 11944: Remove all utf8 filter from templates So I pass the patch to "Patch doesn't apply" status.
Created attachment 28732 [details] [review] Bug 11944: (follow-up) Remove all utf8 filter from templates This patch removes 2 new occurrences introduced by bug 11351 and bug 10493. Signed-off-by: Magnus Enger <digitalutvikling@gmail.com> Removes a nasty "Template process failed: plugin error - EncodeUTF8: plugin not found at /home/magnus/scripts/kohaclone/C4/Templates.pm line 124" from /cgi-bin/koha/admin/itemtypes.pl
It's really a pain to upload/maintain these patches up-to-date on the bug report. I pushed the patches on a remote branch: https://git.biblibre.com/biblibre/kohac/commits/ft/bug_11944 The url of the remote is http://git.biblibre.com/biblibre/kohac.git
commit 35884cf951c2aaa7d57daef264be9eca3c2760bc Author: Jonathan Druart <jonathan.druart@biblibre.com> Date: Mon Jun 9 12:44:35 2014 +0200 Revert "Bug 12279: Diacritics in system preference editor broken" This reverts commit 2fa2480447a58f9d5d3f7834bcfdcf62a0a95df3. Conflicts: admin/preferences.pl It fixes regression introduced by bug 12279.
Sandboxes 6 and 16 use this branch.
*** Bug 10068 has been marked as a duplicate of this bug. ***
A General Bug Sign off Day only for this bug is planned for 23/July/2014
Branch updated against master.
TODO: add an updatedb entry to change the charset collate to utf8_unicode_ci. Somethink like SET foreign_key_checks = 0; foreach table ALTER TABLE `table` CONVERT TO CHARACTER SET utf8 COLLATE utf8_unicode_ci; end SET foreign_key_checks = 1;
Created attachment 29981 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29982 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29983 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29984 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29985 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29986 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with utf-8 char and search for it Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29987 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29988 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29989 [details] [review] Bug 11944: TT Plugins should not encode strings Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29990 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29991 [details] [review] Bug 11944: FIX Error on editing an authority Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29992 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29993 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with utf-8 in password - apply patches - try to log in - change the password - log out - try to log in Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29994 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot). Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29995 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29996 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29997 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29998 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see correctly encoded message Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 29999 [details] [review] Bug 11944: Fix encoding issue on search history Newly introduced by patches from bug 10807, the string was decoded before to be stored in session. To reproduce: Enable history search Go at the OPAC and log out Search for something with utf-8 Go on your search history The search description should be correctly encoded Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 30000 [details] [review] Bug 11944: Remove bad FIXME in guided_report Bug 11679 introduced a FIXME for this but it is wrong. The strings have to be encoded before generating the ods file. However, it is useless to decode column names. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 30001 [details] [review] Bug 11944: Facets and resort Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 30002 [details] [review] Bug 11944: Fix encoding issue in C4::ItemType There is no need to encode strings coming from DB. To reproduce: go on admin/item_circulation_alerts.pl The headers contain bad encoded characters. Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 30003 [details] [review] Bug 11944: (follow-up) Remove all utf8 filter from templates This patch removes 2 new occurrences introduced by bug 11351 and bug 10493. Signed-off-by: Magnus Enger <digitalutvikling@gmail.com> Removes a nasty "Template process failed: plugin error - EncodeUTF8: plugin not found at /home/magnus/scripts/kohaclone/C4/Templates.pm line 124" from /cgi-bin/koha/admin/itemtypes.pl Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 30004 [details] [review] Revert "Bug 12279: Diacritics in system preference editor broken" This reverts commit 2fa2480447a58f9d5d3f7834bcfdcf62a0a95df3. Conflicts: admin/preferences.pl Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org>
Created attachment 30005 [details] [review] Bug 11944: remove url/uri filter from query_cgi We don't need to pass query_cgi through url (or uri) filter since this will turn query string (correctly escaped inside code) back into two-byte escaped string which CGI( -utf8 ) won't turn back into proper utf-8
Created attachment 30006 [details] [review] Bug 11944: decode uri_unescape data This data is passed into form so it needs to be correctly marked as utf-8 if we want form submit to work correctly This change fixes sort issues which use form submit.
Created attachment 30012 [details] [review] Bug 11944: Convert DB tables to utf8_unicode_ci
The branch kohac/ft/bug_11944 is set to master + all patches from this report. The 2 sandboxes 6 and 16 are up-to-date from this branches.
(In reply to Jonathan Druart from comment #235) > Created attachment 30012 [details] [review] [review] > Bug 11944: Convert DB tables to utf8_unicode_ci You are also doing the same at install time?
Created attachment 30016 [details] [review] Bug 11944: Convert DB tables to utf8_unicode_ci
(In reply to M. de Rooy from comment #237) > (In reply to Jonathan Druart from comment #235) > > Created attachment 30012 [details] [review] [review] [review] > > Bug 11944: Convert DB tables to utf8_unicode_ci > > You are also doing the same at install time? Done in the last patch.
Created attachment 30945 [details] [review] Bug 11944: Fix suggestion at the OPAC for perl v5.10 Without this patch and using Perl v5.10, on adding a suggestion at the OPAC, the script explodes with the following error: "cannot decode string with wide characters". With this patch, all is fine using Perl v5.10 and v5.18.
(In reply to Jonathan Druart from comment #240) > Created attachment 30945 [details] [review] [review] > Bug 11944: Fix suggestion at the OPAC for perl v5.10 The last patch fixes the issue raised on bug 12664. The patch is added to the branch kohac/ft/bug_11944. Sandboxes 6 and 16 are up-to-date.
Created attachment 31299 [details] [review] Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci Some lines didn't match the regex I used!
(In reply to Jonathan Druart from comment #242) > Created attachment 31299 [details] [review] [review] > Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci The remote branch does not contain this patch at the moment.
Last patch push on kohac/ft/bug_11944. This branch is up-to-date against master.
sandboxes 6 and 16 up-to-date.
Agreed in Dev Meeting of 02/09/2014 to set status to Signed off.. Branch still maintained by Jonathan is the most up to date patch set. We will add Zeno and Paula's signoff attribute at the last minute ;)
Comment on attachment 31299 [details] [review] Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci Review of attachment 31299 [details] [review]: ----------------------------------------------------------------- First QA point, as mentioned in the Dev meeting, I believe we should jump straight to utf8mb4 for mysql encoding rather than utf8. Should be a trivial change to the last patch. I can confirm that ptfs-europe have been running utf8mb4 databases with koha for a number of years without issue. We have never come up against any problems caused by the character number reduction associated with certain datatypes in utf8mb4 mysql.
OK, I'm pretty much happy to pass this all now, with the exception of I'd like the utf8_mb4 follow up for the database as commented above.. or at least some further input regarding it. I know Galen supported the notion, so unless anyone has a reason not to do it, then I would like to see that change. Fantastic work Jonathan, was a long hard slog, but worth it. As a secondary note, Tomas/Galen, did you get anywhere with writing tests to prevent us getting tied in knots again in the future?
Happy to Passed QA. In my extensive testing I could not find any regressions, and the code all comes up to scratch passing qa script and tests all still pass. We still need the unit tests, but as Tomas/Galen promised to get onto these, I don't see them holding up QA on this part of the proceedings. Jonathan, IRC Gang and I all chatted about utf8_mb4 and we ended up realising that Squeeze still ships with a mysql version over 4 yeras old and so does not support the _mb4 encoding. I don't feel there is enough benefit to be had from mb4 to ask Jonathan to add a followup with code to decide encoding based upon distrobution. (Though I'd love to see this some time, and would suggest opening a followup bug to track it) Fantastic work all involved! PS: Only 'passed' in bugzilla status, Jonathan will add the signoff line in the patches for me. This is to 'keep it simple' regarding conflicts and re-basing :)
(In reply to Martin Renvoize from comment #249) > PS: Only 'passed' in bugzilla status, Jonathan will add the signoff line in > the patches for me. This is to 'keep it simple' regarding conflicts and > re-basing :) Done, signed-off-by line added on 29 patches.
C4::Charset::IsStringUTF8ish seems to be failing some tests. My feeling is that utf8::decode should still be used there, but am waiting for Galen's answer (as the one who wrote the piece of code).
commit bfc167d532788154db64c65c04d80ef603fa56a5 Author: Tomas Cohen Arazi <tomascohen@gmail.com> Date: Wed Nov 5 11:59:38 2014 -0300 Bug 11944: revert unneeded IsStringUTF8ish behaviour change Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
commit 8e4f43aee283869dc78991874ea0ac58cdba8ac0 Author: Jonathan Druart <jonathan.druart@biblibre.com> Date: Thu Dec 4 13:13:30 2014 +0100 Bug 11944: Fix problems introduced by bug 11515 The game continue... Patch for bug 11515 introduced regression on this patch set. To reproduce: search with utf8 characters at the opac Test plan: Verify that the issue described on bug 11515 is still fixed and that no regression is found at the OPAC. Note that I am pretty sure this patch is not enough.
sandboxes 6 and 16 updated with last patches.
I've made some tests on the two sandboxes 6 and 16. The issue described on bug 11515 is still fixed: everything is OK imo.
Guys, can u please take a look at the tests from 13264? It seems that the latest refactoring of the tests yield a situation where the tests don't pass.
Tried to apply this for testing the tests, but it won't apply. The branch also seems outdated. Can you please rebase?
Branch rebased against master.
Patches pushed to master. Thanks Jonthan and everyone involved. Hopefully no problems will arise, keep your eyes open to encoding issues!
(In reply to Tomás Cohen Arazi from comment #259) > Patches pushed to master. > > Thanks Jonthan and everyone involved. > Hopefully no problems will arise, keep your eyes open to encoding issues! Great work, guys. This has been a lot of work! So far I did not spot anything while testing with Latin and Chinese chars. Except one small thing: Frédéric Demians on the about page, timeline tab (reading the history txt file).
(In reply to M. de Rooy from comment #260) > Frédéric Demians on the about page, timeline tab (reading the history txt > file). See bug 13579 for a fix.
I am going to reupload on this bug report the 31 patches pushed into master, for the history and/or backport them.
Created attachment 35223 [details] [review] Bug 11944: Remove all utf8 filter from templates This patch - removes all html_entity usages in tt file which hide utf8 bugs - removes all encode utf8 in tt plugins because we should get correctly marked data from DBIC and other sources directly (cf plugin EncodeUTF8 used in renew.tt) - adds some cleanup in C4::Templates::output: we now use perl utf8 file handler output so we don't need to decode tt variables manually. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35224 [details] [review] Bug 11944: replace use of utf8 with Encode See the wiki page for the explanation. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35225 [details] [review] Bug 11944: Encoding UTF8 in C4::Templates Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35226 [details] [review] Bug 11944: Set ImplicitUnicode for YAML Before this patch, string for yaml files used in preferences.pl were broken. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35227 [details] [review] Bug 11944: Library name variable should be decoded Test on preferences.pl and on some others pages when mysql is used to store session. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35228 [details] [review] Bug 11944: replace uri_escape with uri_escape_utf8 everywhere To reproduce, edit, index notice with utf-8 char and search for it Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35229 [details] [review] Bug 11944: Never use decode/encode in TT Plugins The EncodeUTF8 TT plugin is useless, we should NEVER use encode in templates. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35230 [details] [review] Bug 11944: Remove obsolete encoding routines We don't need to decode manually which provide nice performance boost. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35231 [details] [review] Bug 11944: TT Plugins should not encode strings Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35232 [details] [review] Bug 11944: Some refresh don't indicate the charset to use This patch replaces occurrences of META HTTP-EQUIV=Refresh with a redirection. Note: The list of modified files has been got with the following grep: git grep Content-Type | grep -v -i utf | grep Refresh | grep -v translator | grep -v \.js Test plan: 1) admin/auth_tag_structure.pl - Go on an authority MARC framework for an authority type (admin/auth_tag_structure.pl?authtypecode=CO for instance) - Click on the "Add a new tag" button - Fill fields - Save - You should be redirected on the auth_tag_structure.pl page - Try to modify an existing tag - The redirection should be done 2) admin/auth_subfields_structure.pl - From the previous page, click on a "subfields" link - Edit/Add a new subfields - Save - You should be redirected to the subfield list - Delete a subfield - You should be redirected to the subfield list 3) admin/authorised_values.pl - Add/Edit an authorised value - Save - You should be redirected to the authorised value list (with the right category selected) 4) admin/categorie.pl - Add/Edit a patron category - Save - You should be redirected to the category list - Delete a patron category - You should be redirected to the category list 5) admin/koha2marclinks.pl - Go on the Koha to MARC mapping configuration page - Edit a field - Map a field and click on "OK" - You should be redirected to the list 6) admin/marc_subfields_structure.pl - Go on the MARC frameworks page - Click on "MARC structure" for one on the list - Click on "Subfields" for one on the list - Add/Edit a subfield - Save - You should be redirected to the subfield list 7) admin/systempreferences.pl - Go on the system preferences admin page - Click on the "Local use" tab - Add/Edit a pref - Save - You shold be redirected to the Local use preference list 8) serials/subscription-detail.pl - Go on a subscription detail page - Delete the subscription (using Edit > Delete subscription) - You should be redirected on the serials home page Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35233 [details] [review] Bug 11944: FIX Error on editing an authority Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35234 [details] [review] Bug 11944: Fix encoding issue on searching The search patch should fix non-latin character searches. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35235 [details] [review] Bug 11944: Authentication The password should be encoded before hashing. Test plan: - Before applying the patch, create a user with utf-8 in password - apply patches - try to log in - change the password - log out - try to log in Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35236 [details] [review] Bug 11944: Fix fund names encoding on the parcel page On the parcel page, the subtotal values are stored in an hashref. The key was the fund name. The fund name being encoded in utf8, the hashref was broken. This patch replaces this logic creating a key based on an integer. The budget name is now stored in a new "budget_name" key. Note: It also fixes a small ergonomic issue (a td was missing in the tfoot). Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35237 [details] [review] Bug 11944: Fix encoding issue on adding a subscription If frequencies or numbering patterns contain non-latin characters, so output was broken. To reproduce: - Create a frequency and a numbering pattern with non-latin characters - Add a new subscription using these values - The screen should not contain bad encoded characters. Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35238 [details] [review] Bug 11944: use CGI( -utf8 ) everywhere Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35239 [details] [review] Bug 11944: Fix encoding on sending emails This patch fixes 2 places where mails were badly encoded: 1/ At the opac and the intranet, on sending baskets 2/ At the opac and the intranet, on sending shelf/list Test plan: Shelf/List: - Create a list with non-latin characters in the name. - Add some items containing non-latin characters in their content. - Send the list by email Basket/Cart: - Add some items containing non-latin characters in their content to your cart.. - Send the cart by email Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35240 [details] [review] Bug 11944: Fix encoding issue for the "Did you mean" feature Test plan: 1/ Go on admin/didyoumean.pl and enable the feature for the OPAC 2/ Generate and use the french templates 3/ Launch a search at the OPAC 4/ You should see correctly encoded message Signed-off-by: Paola Rossi <paola.rossi@cineca.it> Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35241 [details] [review] Bug 11944: Fix encoding issue on search history Newly introduced by patches from bug 10807, the string was decoded before to be stored in session. To reproduce: Enable history search Go at the OPAC and log out Search for something with utf-8 Go on your search history The search description should be correctly encoded Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35242 [details] [review] Bug 11944: Remove bad FIXME in guided_report Bug 11679 introduced a FIXME for this but it is wrong. The strings have to be encoded before generating the ods file. However, it is useless to decode column names. Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35243 [details] [review] Bug 11944: Fix encoding issue in C4::ItemType There is no need to encode strings coming from DB. To reproduce: go on admin/item_circulation_alerts.pl The headers contain bad encoded characters. Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35244 [details] [review] Bug 11944: (follow-up) Remove all utf8 filter from templates This patch removes 2 new occurrences introduced by bug 11351 and bug 10493. Signed-off-by: Magnus Enger <digitalutvikling@gmail.com> Removes a nasty "Template process failed: plugin error - EncodeUTF8: plugin not found at /home/magnus/scripts/kohaclone/C4/Templates.pm line 124" from /cgi-bin/koha/admin/itemtypes.pl Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35245 [details] [review] Revert "Bug 12279: Diacritics in system preference editor broken" This reverts commit 2fa2480447a58f9d5d3f7834bcfdcf62a0a95df3. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com> Conflicts: admin/preferences.pl Signed-off-by: Dobrica Pavlinusic <dpavlin@rot13.org> http://bugs.koha-community.org/show_bug.cgi?id=11944 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 35246 [details] [review] Bug 11944: remove url/uri filter from query_cgi We don't need to pass query_cgi through url (or uri) filter since this will turn query string (correctly escaped inside code) back into two-byte escaped string which CGI( -utf8 ) won't turn back into proper utf-8 Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35247 [details] [review] Bug 11944: decode uri_unescape data This data is passed into form so it needs to be correctly marked as utf-8 if we want form submit to work correctly This change fixes sort issues which use form submit. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35248 [details] [review] Bug 11944: Convert DB tables to utf8_unicode_ci Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35249 [details] [review] Bug 11944: Fix suggestion at the OPAC for perl v5.10 Without this patch and using Perl v5.10, on adding a suggestion at the OPAC, the script explodes with the following error: "cannot decode string with wide characters". With this patch, all is fine using Perl v5.10 and v5.18. Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35250 [details] [review] Bug 11944: (follow-up) Convert DB tables to utf8_unicode_ci Some lines didn't match the regex I used! Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35251 [details] [review] Bug 11944: revert unneeded IsStringUTF8ish behaviour change Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35252 [details] [review] Bug 11944: Fix problems introduced by bug 11515 The game continue... Patch for bug 11515 introduced regression on this patch set. To reproduce: search with utf8 characters at the opac Test plan: Verify that the issue described on bug 11515 is still fixed and that no regression is found at the OPAC. Note that I am pretty sure this patch is not enough. Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
Created attachment 35253 [details] [review] Bug 11944: (RM followup) Missing dependency in C4::Context Signed-off-by: Tomas Cohen Arazi <tomascohen@gmail.com>
*** Bug 10075 has been marked as a duplicate of this bug. ***
This bug is resolved already. The major refactoring it involved is already in 3.20 and further bugs / fixes should belong to their own bug reports.