Bug 34745

Summary: ThingISBN broken: User agent blocked
Product: Koha Reporter: Owen Leonard <oleonard>
Component: Architecture, internals, and plumbingAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: fridolin.somers
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=3030
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Bug 34745: FOR TESTING - ThingISBN broken: User agent blocked

Description Owen Leonard 2023-09-08 14:39:27 UTC
Requests to LibraryThing's ThingISBN service are being blocked. Koha tries to access a url like this:

https://www.librarything.com/api/thingISBN/0439139600

...which opens fine in a browser, but triggers the response "WARNING: URL Request Failed 403 Forbidden" when Koha tries it.

Koha is currently sending the user agent "libwww-perl/6.52" and if I use Firefox's developer tools to load the page with that set as a custom user agent I get a CloudFlare-generated "access denied" message.

I see other CloudFlare customers complaining about this, so it may just be a default. Should we ask LibraryThing to change their configuration? Should we send a less obtrusive user agent with the request?
Comment 1 Owen Leonard 2023-09-08 14:53:58 UTC
Created attachment 155436 [details] [review]
Bug 34745: FOR TESTING - ThingISBN broken: User agent blocked

This patch corrects the LibraryThing URL to use https instead of http
and adds a custom agent string to the initialization of LWP::UserAgent:
"Mozilla/5.0"

This is a test to confirm that the a different user agent string might
allow the ThingISBN to work again.
Comment 2 Fridolin Somers 2024-08-06 12:37:20 UTC
Currently opening in a browser shows a page :
"Sorry, you have been blocked"
Comment 3 Owen Leonard 2024-08-06 13:46:54 UTC
LibraryThing seems to have converted this service to an authenticated API. I understand why, but I'm curious why it was done with no apparent announcement.

Also, "LibraryThing's APIs are currently disabled until further notice", posted 2022-05-16: https://wiki.librarything.com/index.php/LibraryThing_APIs
Comment 4 Owen Leonard 2024-08-06 13:50:25 UTC
Okay, not exactly authenticated... Just requires the addition of a token. And it seems to be working at the moment.