Bug 21848 - Resolve unac_string warning from Circulation.t
Summary: Resolve unac_string warning from Circulation.t
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low minor (vote)
Assignee: Marcel de Rooy
QA Contact: Testopia
URL:
Keywords:
Depends on: 19936
Blocks:
  Show dependency treegraph
 
Reported: 2018-11-16 08:03 UTC by Marcel de Rooy
Modified: 2020-01-06 20:15 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 21848: Handle encoding when calling Text::Unaccent::unac_string (2.65 KB, patch)
2018-11-16 08:45 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21848: Remove Text::Unaccent from C4::Members (801 bytes, patch)
2018-11-16 08:45 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21848: Counterpatch to resolve unac_string warning (801 bytes, patch)
2018-12-13 12:09 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21848: Resolve unac_string warning from Circulation.t (815 bytes, patch)
2018-12-14 09:34 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21848: Remove Text::Unaccent from C4::Members (801 bytes, patch)
2018-12-14 09:34 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 21848: Resolve unac_string warning from Circulation.t (856 bytes, patch)
2018-12-14 15:09 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 21848: Remove Text::Unaccent from C4::Members (842 bytes, patch)
2018-12-14 15:09 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Marcel de Rooy 2018-11-16 08:03:23 UTC
Triggered by running Circulation.t on bug 18677:
t/db_dependent/Circulation.t .. 1/121 
unac_string: Invalid or incomplete multibyte or wide character
unac_string: Invalid or incomplete multibyte or wide character

Note that Text::Unaccent says:
Return the unaccented equivalent of the string $string. The character set of $string is specified by the $charset argument. The returned string is coded using the same character set.

The only place we call unac_string is: Koha::Patron->generate_userid:
    $userid = unac_string('utf-8',$userid);
Since we say to unac_string that $userid is utf-8, we should first encode it and the result we should decode again for further handling.
Comment 1 Marcel de Rooy 2018-11-16 08:45:00 UTC
Created attachment 82384 [details] [review]
Bug 21848: Handle encoding when calling Text::Unaccent::unac_string

Resolve warnings like:
    unac_string: Invalid or incomplete multibyte or wide character

Note that Text::Unaccent says:
    Return the unaccented equivalent of the string $string. The character set of $string is specified by the $charset argument. The returned string is coded using the same character set.

The only place we call unac_string is: Koha::Patron->generate_userid:
    $userid = unac_string('utf-8',$userid);
Since we say to unac_string that $userid is utf-8, we should first encode it and the result we should decode again for further handling.

Test plan:
[1] Run t/db_dependent/Koha/Patrons.t
[2] Run t/db_dependent/Circulation.t (before this patch it gave a few warns)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 2 Marcel de Rooy 2018-11-16 08:45:06 UTC
Created attachment 82385 [details] [review]
Bug 21848: Remove Text::Unaccent from C4::Members

At refactoring time the unac_string call was moved to Koha::Patron.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 3 Marcel de Rooy 2018-11-16 08:45:54 UTC
Signing off in stream of 18677
Comment 4 Marcel de Rooy 2018-11-16 08:46:39 UTC
Tomas: Please QA. It is all about tomasito ;)
Comment 5 Jonathan Druart 2018-12-12 15:50:00 UTC
Marcel, do you understand why we do not get the warning from Patrons.t?
Comment 6 Jonathan Druart 2018-12-12 15:51:33 UTC
I think that we should add a test to show that we removed the warning.
Comment 7 Marcel de Rooy 2018-12-13 07:51:03 UTC
(In reply to Jonathan Druart from comment #5)
> Marcel, do you understand why we do not get the warning from Patrons.t?

Since we do not use special chars in the names there.
Which we did in Circulation.t. At least I see:

    my %expired_borrower_data = (
        firstname =>  'Ça',
        surname => 'Glisse',
        categorycode => $patron_category->{categorycode},
        branchcode => $branch,
        dateexpiry => dt_from_string->subtract( months => 1 ),
    );
[…]
Koha::Patron->new(\%expired_borrower_data)->store->borrowernumber;


Yeah, renaming Ça is enough to get rid of the warn.
Comment 8 Marcel de Rooy 2018-12-13 12:05:16 UTC
Hmm. Something strange going on here.

Note that adding use utf8; to Circulation.t also resolves the warning.

Having trouble to catch the warn with warning_like and its friends.

Should we just add "use utf8;" and leave Patron.pm alone ?
Comment 9 Marcel de Rooy 2018-12-13 12:09:58 UTC
Created attachment 83146 [details] [review]
Bug 21848: Counterpatch to resolve unac_string warning

Just add "use utf8".

Test plan:
Prove Circulation.t without warnings.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 David Cook 2018-12-13 23:23:37 UTC Comment hidden (obsolete)
Comment 11 Marcel de Rooy 2018-12-14 09:34:37 UTC
Created attachment 83210 [details] [review]
Bug 21848: Resolve unac_string warning from Circulation.t

Just add "use utf8".

Test plan:
Run Circulation.t; verify there are no warnings.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 12 Marcel de Rooy 2018-12-14 09:34:42 UTC
Created attachment 83211 [details] [review]
Bug 21848: Remove Text::Unaccent from C4::Members

At refactoring time the unac_string call was moved to Koha::Patron.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 13 Marcel de Rooy 2018-12-14 09:36:05 UTC
Jonathan: I decided to simplify here.
Comment 14 Jonathan Druart 2018-12-14 15:09:08 UTC
Created attachment 83217 [details] [review]
Bug 21848: Resolve unac_string warning from Circulation.t

Just add "use utf8".

Test plan:
Run Circulation.t; verify there are no warnings.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 15 Jonathan Druart 2018-12-14 15:09:12 UTC
Created attachment 83218 [details] [review]
Bug 21848: Remove Text::Unaccent from C4::Members

At refactoring time the unac_string call was moved to Koha::Patron.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 16 Nick Clemens 2018-12-20 00:56:51 UTC
Awesome work all!

Pushed to master for 19.05
Comment 17 Martin Renvoize 2018-12-20 10:00:39 UTC
Pushed to 18.11.x for 18.11.01
Comment 18 Lucas Gass 2018-12-31 17:06:55 UTC
backported to 18.05 for 18.05.08
Comment 19 Fridolin Somers 2019-01-02 10:46:59 UTC
Pushed to 17.11.x for 17.11.14
Comment 20 Fridolin Somers 2019-01-02 15:26:15 UTC
Ah brakes Jenkins.

In the end the warn does not exist in 17.11.x.

Caused by Bug 19936 who moved generate_userid to Koha::Patron.