| Summary: | Add user-agent to SUSHI outgoing requests | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Pedro Amorim <pedro.amorim> | 
| Component: | ERM | Assignee: | Pedro Amorim <pedro.amorim> | 
| Status: | CLOSED FIXED | QA Contact: | Nick Clemens (kidclamp) <nick> | 
| Severity: | major | ||
| Priority: | P5 - low | CC: | fridolin.somers, jonathan.druart, jonathan.field, lucas, martin.renvoize, matt.blenkinsop, michaela.sieber, pedro.amorim, wainuiwitikapark | 
| Version: | unspecified | ||
| Hardware: | All | ||
| OS: | All | ||
| GIT URL: | Initiative type: | --- | |
| Sponsorship status: | --- | Crowdfunding goal: | 0 | 
| Patch complexity: | --- | Documentation contact: | |
| Documentation submission: | Text to go in the release notes: | ||
| Version(s) released in: | 24.11.00,24.05.04,23.11.10 | Circulation function: | |
| Attachments: | Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: Add user-agent for SUSHI outgoing requests Bug 37308: (QA follow-up) Tidy code | ||
| 
        
          Description
        
        
          Pedro Amorim
        
        
        
        
          2024-07-10 15:06:56 UTC
        
       Created attachment 168746 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Created attachment 168750 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Pedro Amorim <pedro.amorim@ptfs-europe.com> Created attachment 168751 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Can you explain a bit more about this bug and who might be affected? (In reply to Katrin Fischer from comment #4) > Can you explain a bit more about this bug and who might be affected? Hi Katrin: On one of our customers, for a specific SUSHI provider, whenever a 'test' or 'harvest' request was performed, that specific SUSHI provider returned the following HTTP::Response message (shortened for brevity): '_content' => 'error code: 1010', '_request' => bless( { ... '_method' => 'GET', ... }, 'HTTP::Request' ), '_protocol' => 'HTTP/1.1', '_msg' => 'Forbidden', '_headers' => bless( { ... 'server' => 'cloudflare', ... }, 'HTTP::Headers' ), '_rc' => '403', '_max_body_size' => undef }, 'HTTP::Response' ); Investigating error CloudFlare's code 1010, we find the following: "Error 1010: The owner of this website has banned your access based on your browser's signature" is often encountered when web scraping using browser automation tools like Puppetter, Playwright or Selenium." A curl to the endpoint directly from the server worked, and accessing the URL directly from the browser also worked. Only the request happening directly from Koha's LWP::UserAgent returned Forbidden 403 with the above HTTP::Response. It looked like that provider was considering a request from Koha to be a bot and blocking it. Everything works fine after applying the current patch. Thanks! This is quite hard to come up with a test plan for as credentials are required for a specific provider, but I'm available for a quick call to demonstrate the issue if required. Adding Nick here as QA contact because I stole this solution from his patch: https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=32995&attachment=151124 Created attachment 170377 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 170378 [details] [review] Bug 37308: Add user-agent for SUSHI outgoing requests Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Created attachment 170379 [details] [review] Bug 37308: (QA follow-up) Tidy code Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Pushed for 24.11! Well done everyone, thank you! Backported to 24.05.x for upcoming 24.05.04 Pushed to 23.11.x for 23.11.10 Not backporting to 23.05.x unless requested |