Summary: | Add user-agent to SUSHI outgoing requests | ||
---|---|---|---|
Product: | Koha | Reporter: | Pedro Amorim <pedro.amorim> |
Component: | ERM | Assignee: | Pedro Amorim <pedro.amorim> |
Status: | RESOLVED FIXED | QA Contact: | Nick Clemens (kidclamp) <nick> |
Severity: | major | ||
Priority: | P5 - low | CC: | fridolin.somers, jonathan.druart, jonathan.field, lucas, martin.renvoize, matt.blenkinsop, michaela.sieber, pedro.amorim, wainuiwitikapark |
Version: | unspecified | ||
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | --- |
Documentation contact: | Documentation submission: | ||
Text to go in the release notes: | Version(s) released in: |
24.11.00,24.05.04,23.11.10
|
|
Circulation function: | |||
Attachments: |
Bug 37308: Add user-agent for SUSHI outgoing requests
Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: (QA follow-up) Tidy code |
Description
Pedro Amorim
2024-07-10 15:06:56 UTC
Created attachment 168746 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Created attachment 168750 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com> Created attachment 168751 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Can you explain a bit more about this bug and who might be affected? (In reply to Katrin Fischer from comment #4) > Can you explain a bit more about this bug and who might be affected? Hi Katrin: On one of our customers, for a specific SUSHI provider, whenever a 'test' or 'harvest' request was performed, that specific SUSHI provider returned the following HTTP::Response message (shortened for brevity): '_content' => 'error code: 1010', '_request' => bless( { ... '_method' => 'GET', ... }, 'HTTP::Request' ), '_protocol' => 'HTTP/1.1', '_msg' => 'Forbidden', '_headers' => bless( { ... 'server' => 'cloudflare', ... }, 'HTTP::Headers' ), '_rc' => '403', '_max_body_size' => undef }, 'HTTP::Response' ); Investigating error CloudFlare's code 1010, we find the following: "Error 1010: The owner of this website has banned your access based on your browser's signature" is often encountered when web scraping using browser automation tools like Puppetter, Playwright or Selenium." A curl to the endpoint directly from the server worked, and accessing the URL directly from the browser also worked. Only the request happening directly from Koha's LWP::UserAgent returned Forbidden 403 with the above HTTP::Response. It looked like that provider was considering a request from Koha to be a bot and blocking it. Everything works fine after applying the current patch. Thanks! This is quite hard to come up with a test plan for as credentials are required for a specific provider, but I'm available for a quick call to demonstrate the issue if required. Adding Nick here as QA contact because I stole this solution from his patch: https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=32995&attachment=151124 Created attachment 170377 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 170378 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 170379 [details] [review] Bug 37308: (QA follow-up) Tidy code Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Pushed for 24.11! Well done everyone, thank you! Backported to 24.05.x for upcoming 24.05.04 Pushed to 23.11.x for 23.11.10 Not backporting to 23.05.x unless requested |