Bug 21122

Summary: Make check-url-quick.pl handle utf8 characters in urls gracefuly
Product: Koha Reporter: Liz Rea <liz>
Component: Command-line UtilitiesAssignee: Liz Rea <liz>
Status: CLOSED FIXED QA Contact: Jonathan Druart <jonathan.druart>
Severity: major    
Priority: P5 - low CC: fridolin.somers, jonathan.druart, jose-mario.monteiro-santos, martin.renvoize, robin, tomascohen
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 21122: check-url-quick dies on utf8 characters in URL's
Bug 21122: check-url-quick dies on utf8 characters in URL's
Bug 21122: check-url-quick dies on utf8 characters in URL's
Bug 21122: Fix check-url-quick when utf8 characters in URL's

Description Liz Rea 2018-07-26 03:11:15 UTC
Example URL: http://some.nonexistent.tld/MāoriWomenAotearoa.pdf

Job dies with "Wide character in print" 

Solution seems to be to encode the URL.
Comment 1 Liz Rea 2018-07-26 04:28:02 UTC
Created attachment 77277 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as: http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.
Comment 2 Owen Leonard 2018-07-26 14:00:29 UTC
Created attachment 77289 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>
Comment 3 Jonathan Druart 2018-07-31 15:31:44 UTC
I got worst result without the patch:

15      http://some.nonexistent.tld/MāoriWomenAotearoa.pdf      595 No such device or address
EV: error in callback (ignoring): Wide character in subroutine entry at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 1007.

Wide character in print at misc/cronjobs/check-url-quick.pl line 67.
15      http://ko.wikipedia.org/wiki/위키백과:대문      596 Connection timed out
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
EV: error in callback (ignoring): Wide character in syswrite at /usr/lib/x86_64-linux-gnu/perl5/5.20/AnyEvent/Handle.pm line 2285.
[...infinite loop...]
Comment 4 Jonathan Druart 2018-07-31 15:33:31 UTC
Created attachment 77338 [details] [review]
Bug 21122: check-url-quick dies on utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 5 Jonathan Druart 2018-07-31 15:34:08 UTC
Created attachment 77339 [details] [review]
Bug 21122: Fix check-url-quick when utf8 characters in URL's

To test:

Make a record with a URL that has a UTF8 character, such as:
http://some.nonexistent.tld/MāoriWomenAotearoa.pdf
Run the check-url-quick.pl job, notice it dies at that URL
Apply this patch
Test again, it should work.

Signed-off-by: Owen Leonard <oleonard@myacpl.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 6 Tomás Cohen Arazi 2018-08-02 13:25:03 UTC
Pushed to master, thanks!
Comment 7 Martin Renvoize 2018-08-03 14:21:59 UTC
Pushed to 18.05.x for 18.05.03.
Comment 8 Fridolin Somers 2018-08-28 13:12:10 UTC
Pushed to 17.11.x for 17.11.09
Comment 9 Liz Rea 2018-08-28 23:04:43 UTC
The UTF8 URLS of the world thank you all kindly <3
Comment 10 José-Mario Monteiro-Santos 2019-03-19 18:41:01 UTC
I'm not quite sure if I can ask about this here, but I've been having problems when urls contain characters such as é or à. I understand that this patch makes it so check-url-quick doesn't explode when it encounters such characters, but when the http request is sent they're encoded differently and are marked as broken links (é turns into é for example). Is this related to this bug ?
Comment 11 Jonathan Druart 2019-04-29 02:30:00 UTC
(In reply to José-Mario Monteiro-Santos from comment #10)
> I'm not quite sure if I can ask about this here, but I've been having
> problems when urls contain characters such as é or à. I understand that this
> patch makes it so check-url-quick doesn't explode when it encounters such
> characters, but when the http request is sent they're encoded differently
> and are marked as broken links (é turns into é for example). Is this
> related to this bug ?

No, or I do not understand what you meant exactly.

I have tried with 856$u = https://en.wikipedia.org/wiki/Māori, then run the script with --verbose:
1       https://en.wikipedia.org/wiki/Māori     200 OK                        

Looks ok for me.