Bug 21122 - Make check-url-quick.pl handle utf8 characters in urls gracefuly
Summary: Make check-url-quick.pl handle utf8 characters in urls gracefuly
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Liz Rea
QA Contact: Jonathan Druart
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-07-26 03:11 UTC by Liz Rea
Modified: 2020-01-06 20:15 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 21122: check-url-quick dies on utf8 characters in URL's (1.36 KB, patch)
2018-07-26 04:28 UTC, Liz Rea
Details | Diff | Splinter Review
Bug 21122: check-url-quick dies on utf8 characters in URL's (1.41 KB, patch)
2018-07-26 14:00 UTC, Owen Leonard
Details | Diff | Splinter Review
Bug 21122: check-url-quick dies on utf8 characters in URL's (1.49 KB, patch)
2018-07-31 15:33 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 21122: Fix check-url-quick when utf8 characters in URL's (1.49 KB, patch)
2018-07-31 15:34 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Liz Rea 2018-07-26 03:11:15 UTC
Example URL: http://some.nonexistent.tld/MāoriWomenAotearoa.pdf

Job dies with "Wide character in print" 

Solution seems to be to encode the URL.
Comment 1 Liz Rea 2018-07-26 04:28:02 UTC
Created attachment 77277 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as: http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.
Comment 2 Owen Leonard 2018-07-26 14:00:29 UTC
Created attachment 77289 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 3 Jonathan Druart 2018-07-31 15:31:44 UTC
I got worst result without the patch:

15      http://some.nonexistent.tld/MāoriWomenAotearoa.pdf      595 No such device or address
EV: error in callback (ignoring): Wide character in subroutine entry at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1007.

Wide character in print at misc/cronjobs/check-url-quick.pl line 67.
15      http://ko.wikipedia.org/wiki/위키백과:대문      596 Connection timed out
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
[...infinite loop...]
Comment 4 Jonathan Druart 2018-07-31 15:33:31 UTC
Created attachment 77338 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 5 Jonathan Druart 2018-07-31 15:34:08 UTC
Created attachment 77339 [details] [review]
Bug 21122: Fix check-url-quick when utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 6 Tomás Cohen Arazi 2018-08-02 13:25:03 UTC
Pushed to master, thanks!
Comment 7 Martin Renvoize 2018-08-03 14:21:59 UTC
Pushed to 18.05.x for 18.05.03.
Comment 8 Fridolin Somers 2018-08-28 13:12:10 UTC
Pushed to 17.11.x for 17.11.09
Comment 9 Liz Rea 2018-08-28 23:04:43 UTC
The UTF8 URLS of the world thank you all kindly <3
Comment 10 José-Mario Monteiro-Santos 2019-03-19 18:41:01 UTC
I'm not quite sure if I can ask about this here, but I've been having problems when urls contain characters such as é or à. I understand that this patch makes it so check-url-quick doesn't explode when it encounters such characters, but when the http request is sent they're encoded differently and are marked as broken links (é turns into é for example). Is this related to this bug ?
Comment 11 Jonathan Druart 2019-04-29 02:30:00 UTC
(In reply to José-Mario Monteiro-Santos from comment #10)
> I'm not quite sure if I can ask about this here, but I've been having
> problems when urls contain characters such as é or à. I understand that this
> patch makes it so check-url-quick doesn't explode when it encounters such
> characters, but when the http request is sent they're encoded differently
> and are marked as broken links (é turns into é for example). Is this
> related to this bug ?

No, or I do not understand what you meant exactly.

I have tried with 856$u = https://en.wikipedia.org/wiki/Māori, then run the script with --verbose:
1       https://en.wikipedia.org/wiki/Māori     200 OK                        

Looks ok for me.