Bug 37308 - Add user-agent to SUSHI outgoing requests
Summary: Add user-agent to SUSHI outgoing requests
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: ERM (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Pedro Amorim
QA Contact: Nick Clemens (kidclamp)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2024-07-10 15:06 UTC by Pedro Amorim
Modified: 2024-07-17 15:46 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 37308: Add user-agent for SUSHI outgoing requests (1.30 KB, patch)
2024-07-10 15:07 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 37308: Add user-agent for SUSHI outgoing requests (1.36 KB, patch)
2024-07-10 15:54 UTC, Pedro Amorim
Details | Diff | Splinter Review
Bug 37308: Add user-agent for SUSHI outgoing requests (1.35 KB, patch)
2024-07-10 15:55 UTC, Pedro Amorim
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Pedro Amorim 2024-07-10 15:06:56 UTC
Some SUSHI endpoints are returning Forbidden 403 by Cloudflare.
Adding the same pattern utilized in Overdrive.pm has fixed the issue:
$ua->agent( 'Koha/'.Koha::version() );
Comment 1 Pedro Amorim 2024-07-10 15:07:28 UTC
Created attachment 168746 [details] [review]
Bug 37308: Add user-agent for SUSHI outgoing requests
Comment 2 Pedro Amorim 2024-07-10 15:54:13 UTC Comment hidden (obsolete)
Comment 3 Pedro Amorim 2024-07-10 15:55:03 UTC
Created attachment 168751 [details] [review]
Bug 37308: Add user-agent for SUSHI outgoing requests

Signed-off-by: Belal Ahmadi <belal.ahmadi@uwl.ac.uk>
Comment 4 Katrin Fischer 2024-07-11 08:20:09 UTC
Can you explain a bit more about this bug and who might be affected?
Comment 5 Pedro Amorim 2024-07-11 08:35:13 UTC
(In reply to Katrin Fischer from comment #4)
> Can you explain a bit more about this bug and who might be affected?

Hi Katrin:
On one of our customers, for a specific SUSHI provider, whenever a 'test' or 'harvest' request was performed, that specific SUSHI provider returned the following HTTP::Response message (shortened for brevity):

 '_content' => 'error code: 1010',
 '_request' => bless( {
                       ...
                        '_method' => 'GET',
                        ...
                      }, 'HTTP::Request' ),
 '_protocol' => 'HTTP/1.1',
 '_msg' => 'Forbidden',
 '_headers' => bless( {
                        ...
                        'server' => 'cloudflare',
                        ...
                      }, 'HTTP::Headers' ),
 '_rc' => '403',
 '_max_body_size' => undef
}, 'HTTP::Response' );

Investigating error CloudFlare's code 1010, we find the following:
"Error 1010: The owner of this website has banned your access based on your browser's signature" is often encountered when web scraping using browser automation tools like Puppetter, Playwright or Selenium."

A curl to the endpoint directly from the server worked, and accessing the URL directly from the browser also worked. Only the request happening directly from Koha's LWP::UserAgent returned Forbidden 403 with the above HTTP::Response. It looked like that provider was considering a request from Koha to be a bot and blocking it.
Everything works fine after applying the current patch.
Comment 6 Katrin Fischer 2024-07-11 08:49:04 UTC
Thanks!
Comment 7 Pedro Amorim 2024-07-17 15:46:04 UTC
This is quite hard to come up with a test plan for as credentials are required for a specific provider, but I'm available for a quick call to demonstrate the issue if required.

Adding Nick here as QA contact because I stole this solution from his patch:
https://bugs.koha-community.org/bugzilla3/page.cgi?id=splinter.html&bug=32995&attachment=151124