Bug 21122 - Make check-url-quick.pl handle utf8 characters in urls gracefuly
Summary: Make check-url-quick.pl handle utf8 characters in urls gracefuly
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Liz Rea
QA Contact: Jonathan Druart
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-07-26 03:11 UTC by Liz Rea
Modified: 2018-09-21 09:57 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 21122: check-url-quick dies on utf8 characters in URL's (1.36 KB, patch)
2018-07-26 04:28 UTC, Liz Rea
Details | Diff | Splinter Review
Bug 21122: check-url-quick dies on utf8 characters in URL's (1.41 KB, patch)
2018-07-26 14:00 UTC, Owen Leonard
Details | Diff | Splinter Review
Bug 21122: check-url-quick dies on utf8 characters in URL's (1.49 KB, patch)
2018-07-31 15:33 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 21122: Fix check-url-quick when utf8 characters in URL's (1.49 KB, patch)
2018-07-31 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Liz Rea 2018-07-26 03:11:15 UTC
Example URL: http://some.nonexistent.tld/MāoriWomenAotearoa.pdf

Job dies with "Wide character in print" 

Solution seems to be to encode the URL.
Comment 1 Liz Rea 2018-07-26 04:28:02 UTC
Created attachment 77277 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as: http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.
Comment 2 Owen Leonard 2018-07-26 14:00:29 UTC
Created attachment 77289 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 3 Jonathan Druart 2018-07-31 15:31:44 UTC
I got worst result without the patch:

15      http://some.nonexistent.tld/MāoriWomenAotearoa.pdf      595 No such device or address
EV: error in callback (ignoring): Wide character in subroutine entry at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1007.

Wide character in print at misc/cronjobs/check-url-quick.pl line 67.
15      http://ko.wikipedia.org/wiki/위키백과:대문      596 Connection timed out
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
[...infinite loop...]
Comment 4 Jonathan Druart 2018-07-31 15:33:31 UTC
Created attachment 77338 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 5 Jonathan Druart 2018-07-31 15:34:08 UTC
Created attachment 77339 [details] [review]
Bug 21122: Fix check-url-quick when utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 6 Tomás Cohen Arazi 2018-08-02 13:25:03 UTC
Pushed to master, thanks!
Comment 7 Martin Renvoize 2018-08-03 14:21:59 UTC
Pushed to 18.05.x for 18.05.03.
Comment 8 Fridolin SOMERS 2018-08-28 13:12:10 UTC
Pushed to 17.11.x for 17.11.09
Comment 9 Liz Rea 2018-08-28 23:04:43 UTC
The UTF8 URLS of the world thank you all kindly <3