Bug 39109

Summary: Add throttling / rate limiting to Koha
Product: Koha Reporter: Kyle M Hall (khall) <kyle>
Component: Architecture, internals, and plumbingAssignee: David Cook <dcook>
Status: In Discussion --- QA Contact: Testopia <testopia>
Severity: enhancement    
Priority: P5 - low CC: clemens.tubach, dcook, lukasz.koszyk, m.de.rooy, michaela.sieber, raphael.straub
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=31242
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39435
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Bug 39109: Add Plack middleware to throttle requests

Description Kyle M Hall (khall) 2025-02-12 15:39:05 UTC
Botnets and web spiders can easily bring a Koha instance down. We should implement optional rate limiting in Koha. It appears there are two plack middlewares we could use: https://metacpan.org/pod/Plack::Middleware::Throttle or https://metacpan.org/pod/Plack::Middleware::Throttle::Lite

Throttle supports memcached as a backend, but is more complex. Lite uses an in-memory backend, but has fewer dependencies.
Comment 1 David Cook 2025-02-16 22:38:32 UTC
(In reply to Kyle M Hall (khall) from comment #0)
> Botnets and web spiders can easily bring a Koha instance down. We should
> implement optional rate limiting in Koha. It appears there are two plack
> middlewares we could use:
> https://metacpan.org/pod/Plack::Middleware::Throttle or
> https://metacpan.org/pod/Plack::Middleware::Throttle::Lite
> 
> Throttle supports memcached as a backend, but is more complex. Lite uses an
> in-memory backend, but has fewer dependencies.

I was inspired by Plack::Middleware::Throttle to write my own Koha::Middleware::Throttle. It uses memcached as the storage backend, and its expiring keys mechanism means it is self-managing, which is great.

My version does use some local custom code for managing configuration in a more flexible way, so I'll need to strip that out, but it's still useful without it. Happy to share my work here.
Comment 2 David Cook 2025-02-17 00:18:47 UTC
Created attachment 178130 [details] [review]
Bug 39109: Add Plack middleware to throttle requests

This patch includes a configurable Plack middleware which limits the
number of requests for an IP address range for a moving time window.

For example, 192.168.1.0/24 can only make 5 requests over the last 1 minute.
Comment 3 David Cook 2025-02-17 00:20:24 UTC
Test plan:
1. vi /etc/koha/sites/kohadev/plack.psgi
2. Add the following to the /opac mount point (higher than Koha::Middlware::CSRF but lower than HTTPExceptions) :
        enable "+Koha::Middleware::Throttle",
            debug => 1,
            interface => 'opac',
            paths => [ '^/opac-search', ],
            allow_list => [
                '192.168.1.0/24',
            ],
            request_threshold => 5, #Number of requests to trigger throttle
            last_n_minutes => 1, #Number of minutes to consider when aggregating request counts
            cidr_block => 24, #CIDR mask to use for determing range IP address belongs to (may only be 8, 16, 24, 32)
        ;
3. sudo koha-plack --restart kohadev
4. Do 6 requests to opac-search.pl within 1 minute and see the pretty "You've been throttled" page.
Comment 4 David Cook 2025-02-17 00:22:27 UTC
I have other work to present an interactive challenge for users, but it's not currently compatible with this feature, nor ready to be shared just yet. 

I think a challenge page is better than a blocked page, as it gives legitimate human users an opportunity to get back to searching. But that's a whole other thing.
Comment 5 David Cook 2025-02-17 00:29:03 UTC
I was inspired by Plack::Middleware::Throttle, but I found it to be too limited for our purposes.

You'll likely notice limited use of Koha::Cache, and that's because we don't implement the "incr" method in Koha::Cache, which is vital for multi-process safety in this feature.

--

I think there's lots of room for improvement with this feature, although it is limited in what it can do against distributed botnets and web spiders. 

That said, I just looked at some logs, and I saw it throttled a couple of spiders, so every little bit helps.
Comment 6 Marcel de Rooy 2025-03-19 12:41:23 UTC
Thanks for sharing this. We are currently under fire too :)
Comment 7 Marcel de Rooy 2025-03-19 12:42:56 UTC
I am still thinking about a challenge and some cookie to pass the challenge..
Comment 8 Marcel de Rooy 2025-03-19 13:02:14 UTC
(In reply to David Cook from comment #4)
> I have other work to present an interactive challenge for users, but it's
> not currently compatible with this feature, nor ready to be shared just yet. 
> 
> I think a challenge page is better than a blocked page, as it gives
> legitimate human users an opportunity to get back to searching. But that's a
> whole other thing.

If you submit, please keep me in cc.