Some SUSHI endpoints are returning Forbidden 403 by Cloudflare. Adding the same pattern utilized in Overdrive.pm has fixed the issue: $ua->agent( 'Koha/'.Koha::version() );
Created attachment 168746 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests
Created attachment 168750 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com>
Created attachment 168751 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk>
Can you explain a bit more about this bug and who might be affected?
(In reply to Katrin Fischer from comment #4) > Can you explain a bit more about this bug and who might be affected? Hi Katrin: On one of our customers, for a specific SUSHI provider, whenever a 'test' or 'harvest' request was performed, that specific SUSHI provider returned the following HTTP::Response message (shortened for brevity): '_content' => 'error code: 1010', '_request' => bless( { ... '_method' => 'GET', ... }, 'HTTP::Request' ), '_protocol' => 'HTTP/1.1', '_msg' => 'Forbidden', '_headers' => bless( { ... 'server' => 'cloudflare', ... }, 'HTTP::Headers' ), '_rc' => '403', '_max_body_size' => undef }, 'HTTP::Response' ); Investigating error CloudFlare's code 1010, we find the following: "Error 1010: The owner of this website has banned your access based on your browser's signature" is often encountered when web scraping using browser automation tools like Puppetter, Playwright or Selenium." A curl to the endpoint directly from the server worked, and accessing the URL directly from the browser also worked. Only the request happening directly from Koha's LWP::UserAgent returned Forbidden 403 with the above HTTP::Response. It looked like that provider was considering a request from Koha to be a bot and blocking it. Everything works fine after applying the current patch.
Thanks!
This is quite hard to come up with a test plan for as credentials are required for a specific provider, but I'm available for a quick call to demonstrate the issue if required. Adding Nick here as QA contact because I stole this solution from his patch: https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=32995&attachment=151124
Created attachment 170377 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 170378 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 170379 [details] [review] Bug 37308: (QA follow-up) Tidy code Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Pushed for 24.11! Well done everyone, thank you!
Backported to 24.05.x for upcoming 24.05.04
Pushed to 23.11.x for 23.11.10