As has been noted on bug 25870, there are times where we want to rate limit the REST API.
On Bug 25870, Tomas suggested Apache for rate-limiting, while Kyle suggested Plack middleware to keep the rate-limiting in the Koha app. I'll throw some API Gateway options into the mix: https://konghq.com/ https://www.krakend.io/ (I use Kong for managing complex security requirements around APIs on other projects. I've been wanting to try out KrakenD for a while as it apparently is faster and lighter weight overall.) -- Overall, I'd say that Apache - in its role as reverse proxy to Starman - is the place to do rate limiting. It's almost certainly going to be more efficient than Starman, and there's no point letting in traffic to Starman if we're just going to block it. That said, a quick Google doesn't really show any good Apache options. I think mod_security might be one way which is available but it has a questionable past and future I think.
Maybe it does make sense to put the rate limiting in Starman just to keep things as simple as possible. We could just put it in the builder for "/api/v1/app.pl", so it wouldn't affect the performance of the /opac and /intranet apps. The middlewares Kyle linked do look ancient though. https://www.krakend.io/docs/endpoints/rate-limit/ poses some interesting points about rate limiting methods, especially whether to rate limit by endpoint or rate limit by endpoint * client IP address. One way or another, it would be good to have rate limiting I reckon.
(In reply to David Cook from comment #2) > Maybe it does make sense to put the rate limiting in Starman just to keep > things as simple as possible. Yes, that was the idea. If it's baked in, it's less onerous for libraries. > We could just put it in the builder for "/api/v1/app.pl", so it wouldn't > affect the performance of the /opac and /intranet apps. Is there a benefit to rate limiting everything? It hasn't been an issue lately but back in the day I know we had an issue with "thrashing" where some partner's browsers would DOS a server with requests for reasons we never understood. > > The middlewares Kyle linked do look ancient though. True, but if it works, it doesn't need updated! > https://www.krakend.io/docs/endpoints/rate-limit/ poses some interesting > points about rate limiting methods, especially whether to rate limit by > endpoint or rate limit by endpoint * client IP address. That *is* interesting! I seems more "fair" to do it by endpoint+ip, but really, a system can only handle some much traffic no matter how fair it is. Now I'm going to say it would be nice to have both of those, and for them to each be configurable ;) > One way or another, it would be good to have rate limiting I reckon.
(In reply to Kyle M Hall from comment #3) > (In reply to David Cook from comment #2) > > Maybe it does make sense to put the rate limiting in Starman just to keep > > things as simple as possible. > > Yes, that was the idea. If it's baked in, it's less onerous for libraries. While technically it might not be ideal, I think it would be practical for a FOSS project like Koha for sure. Having something out of the box makes sense. I often think about how X, Y, Z technologies would be great for Koha, but how they're not practical because they are too complex for non-technical Koha adopters... I answered a query about horizontally scaling Koha, and I suppose if you were to horizontally scale Koha, you'd want your rate limiting to be done by your load balancer. But we can cross that bridge when we come to it too. > > We could just put it in the builder for "/api/v1/app.pl", so it wouldn't > > affect the performance of the /opac and /intranet apps. > > Is there a benefit to rate limiting everything? It hasn't been an issue > lately but back in the day I know we had an issue with "thrashing" where > some partner's browsers would DOS a server with requests for reasons we > never understood. If we rate limited everything (without IP checking), I think they'd still DOS Koha just by hitting the global rate limit. We still occasionally have issues where out of control bots (usually benign crawlers) will DOS a Koha instance (not the whole server fortunately) by keeping Starman busy, but we have additional software that finds and blocks those. In theory, we could install fail2ban alongside Koha and point it at Plack logs... (I suppose that is a different kind of rate limiting) > > The middlewares Kyle linked do look ancient though. > > True, but if it works, it doesn't need updated! Haha. That's what I was thinking as well. > > https://www.krakend.io/docs/endpoints/rate-limit/ poses some interesting > > points about rate limiting methods, especially whether to rate limit by > > endpoint or rate limit by endpoint * client IP address. > > That *is* interesting! I seems more "fair" to do it by endpoint+ip, but > really, a system can only handle some much traffic no matter how fair it is. > Now I'm going to say it would be nice to have both of those, and for them to > each be configurable ;) I don't know if you followed the link, but that's what they recommend! "Both and" rather than "Either or". You determine how many simultaneous clients you expect to handle, give each client up to X requests per Y and then have a Z global limit that should be the sum of X for all clients. Probably would need to tweak it a bit from there but not a bad guideline.
> I answered a query about horizontally scaling Koha, and I suppose if you > were to horizontally scale Koha, you'd want your rate limiting to be done by > your load balancer. But we can cross that bridge when we come to it too. Yep, and nothing would prevent you from doing so! Just as any power use could replace apache with nginx or some other frontend. Any rate limiting feature could be turned off or set to an impossibly high threshold. > If we rate limited everything (without IP checking), I think they'd still > DOS Koha just by hitting the global rate limit. Indeed. It's a nontrivial problem, especially when the DOSing pc is inside a library or campus! > We still occasionally have issues where out of control bots (usually benign > crawlers) will DOS a Koha instance (not the whole server fortunately) by > keeping Starman busy, but we have additional software that finds and blocks > those. Ooh, what are you using? > In theory, we could install fail2ban alongside Koha and point it at Plack > logs... (I suppose that is a different kind of rate limiting) That's an interesting idea!
(In reply to Kyle M Hall from comment #5) > Yep, and nothing would prevent you from doing so! Just as any power use > could replace apache with nginx or some other frontend. Any rate limiting > feature could be turned off or set to an impossibly high threshold. Yeah I was thinking the same thing. So long as we make things optional/configurable, it's all good. > > We still occasionally have issues where out of control bots (usually benign > > crawlers) will DOS a Koha instance (not the whole server fortunately) by > > keeping Starman busy, but we have additional software that finds and blocks > > those. > > Ooh, what are you using? A combo of fail2ban and custom scripts. It can always use tweaking! > > In theory, we could install fail2ban alongside Koha and point it at Plack > > logs... (I suppose that is a different kind of rate limiting) > > That's an interesting idea! Right? I feel like it would be tough to get it right though. -- In any case, one of those Plack middlewares for the "api" app in plack.psgi is probably a good place to start? I mean it is better than the current situation at the very least.