In my experience, Bug 39435 works well, but if you only have a few Starman workers it's still a suboptimal use of resources. By doing a bot challenge at the Apache reverse proxy level, we're able to save our Starman resources for when we really need them.
Created attachment 182474 [details] [review] Bug 39906: Add a simple quick Apache-level bot challenge This patch adds a fast simple bot challenge via Apache. To test: 0. Apply the patch 1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/. 2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/. 3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl 4. Note an instant response with a very simple HTML response 5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl 6. Note a very brief loading animation before the OPAC search page appearing 7. Do a search 8. Note no more loading animation appears 9. Close the incognito browser 10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl 11. Do a search via the "Library catalog" search bar 12. Note no loading animation appears
Note that I don't really think this is "Needs Signoff" but it's in a testable state. -- One thing to note... this patch will probably cause our Selenium testing to slow down significantly, so there's probably some tweaks to do here. But this should be a simple enough proof-of-concept.
Nice! What should be the expiration of KOHA_INIT cookie ?
(In reply to Marcel de Rooy from comment #3) > Nice! > What should be the expiration of KOHA_INIT cookie ? I think we probably have to keep it as a session expiry, since we can't do a dynamic calculation here.
(In reply to David Cook from comment #4) > (In reply to Marcel de Rooy from comment #3) > > Nice! > > What should be the expiration of KOHA_INIT cookie ? > > I think we probably have to keep it as a session expiry, since we can't do a > dynamic calculation here. Actually, as I was leaving work yesterday, I was thinking about some load testing I was doing and how I wasn't happy with the KOHA_INIT cookie being set in the Apache. I think it'll be better to move it to the Javascript. I originally was setting the KOHA_INIT cookie when I was sending a 302 in Apache rather than doing an internal redirect. It's not really needed anymore. New patch incoming...
Created attachment 182506 [details] [review] Bug 39906: Add a simple quick Apache-level bot challenge This patch adds a fast simple bot challenge via Apache. To test: 0. Apply the patch 1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/. 2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/. 3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl 4. Note an instant response with a very simple HTML response 5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl 6. Note a very brief loading animation before the OPAC search page appearing 7. Do a search 8. Note no more loading animation appears 9. Close the incognito browser 10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl 11. Do a search via the "Library catalog" search bar 12. Note no loading animation appears
Ok now we get into the fun part... To load test: 0. apt-get install siege 1. siege -d 1 -c 12 -t 300s --no-parser --internet "http://127.0.0.1:8080/cgi-bin/koha/opac-search.pl" 1b. Tweak the concurrent request -c parameter as necessary. If you have a very low spec machine, this may not be a good test (as your machine will be handles you, siege, KTD, etc) 2. Try navigating your KTD OPAC and staff interface 3. Note that it still feels usable (depending on the number of CPUs you have available to KTD of course) 4. sudo tail -f /var/log/apache2/other_vhosts_access.log 5. Note the huge flood of incoming requests from Siege being met with small responses back 6. ss -t -n -l 7. Note that the Recv-Q for 0.0.0.0:8080 is 0 or low (again depends on the number of CPUs you have available
Created attachment 182507 [details] [review] Bug 39906: Add a simple quick Apache-level bot challenge This patch adds a fast simple bot challenge via Apache. To test: 0. Apply the patch 1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/. 2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/. 3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl 4. Note an instant response with a very simple HTML response 5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl 6. Note a very brief loading animation before the OPAC search page appearing 7. Do a search 8. Note no more loading animation appears 9. Close the incognito browser 10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl 11. Do a search via the "Library catalog" search bar 12. Note no loading animation appears
Fixed up the HTTPS check to add "Secure" to the cookie. Not really necessary I guess, but I figure why not.
We can confirm that the solution works as described on a Koha 22.11.26 instance. We cannot confirm yet how effective the solution it, but maybe soon. Some questions that popped up looking at the solution: * Wondering if the cookie value could be validated in some way. Currently we have KOHA_INIT=1 in the cookie. Could 1 be switched to a hash value for example or something else cryptic that is ideally also verified by the server? * We have monitoring that checks if the search is still working. That will also be affected by the challenge on opac-search. Would it be possible to allow certain IPs to get to the page unchallenged?
(In reply to Katrin Fischer from comment #10) > Some questions that popped up looking at the solution: > > * Wondering if the cookie value could be validated in some way. Currently we > have KOHA_INIT=1 in the cookie. Could 1 be switched to a hash value for > example or something else cryptic that is ideally also verified by the > server? Right now, the existence of the KOHA_INIT cookie is all that Apache checks (the value is irrelevant), and Apache is quite limited in what it can do without the use of something like mod_perl or mod_lua. But yeah... there's things that could be done. Rather than checking for the presence of KOHA_INIT in Apache, you could always pass through to the Javascript, and have it do a check. But... then if the check involves an API call that's going to be additional load on the Koha server. It might seem small but when you start scaling up the requests, that's when Starman starts to suffer. In practice, I haven't found a need to validate the contents of the KOHA_INIT cookie, but in the future it might be necessary. An easy one to do could be to actually store the client's IP address in KOHA_INIT. That way, if the KOHA_INIT is sent by a different IP address than the one that generated the cookie, then it would be an easy catch/validation. Other than that... someone could experiment with Apache but they might have to go the mod_perl or mod_lua route. mod_perl would probably make sense for Koha people since we're already familiar with Perl. I use mod_perl a lot in a different non-Koha app, so I can probably help out with that. I suppose one thing that could be experimented with is mod_expr. We could also always add a CGI script, which Apache can invoke independently of Starman. That would have server overheads but wouldn't impact Starman. Not an ideal solution though. On a similar note, we could do a FCGI handler, but then that seems like overkill. Overall, I'd say there are options, but... they're not really necessary at the moment. > * We have monitoring that checks if the search is still working. That will > also be affected by the challenge on opac-search. Would it be possible to > allow certain IPs to get to the page unchallenged? Yes, absolutely. That's what I do locally. All you need to do is add a RewriteCond in that debian/templates/apache-shared-opac-antibot.conf file. For example, if you want to allow everything on 192.168.1.0/24, you'd just add: RewriteCond %{REMOTE_ADDR} !^192\.168\.1 That way, only IP addresses other than that one would trigger the condition.
(In reply to David Cook from comment #11) > An easy one to do could be to actually store the client's IP address in > KOHA_INIT. That way, if the KOHA_INIT is sent by a different IP address than > the one that generated the cookie, then it would be an easy > catch/validation. Remenber that people may change IP while travelling. So you need to decide if the cookie is IP independent or another IP needs a new cookie (probably better: if one bot obtains the cookie, do not allow them all). > We could also always add a CGI script, which Apache can invoke independently > of Starman. That would have server overheads but wouldn't impact Starman. > Not an ideal solution though. Ultimately I ended up with an in-between container for those reasons..
(In reply to Marcel de Rooy from comment #12) > Remenber that people may change IP while travelling. So you need to decide > if the cookie is IP independent or another IP needs a new cookie (probably > better: if one bot obtains the cookie, do not allow them all). Yeah, I've thought about that scenario as well. Or they might be using a VPN or HTTP forward proxy that has a range of outbound IP addresses. However, since is just needed for a top-level navigation, it doesn't really matter if we challenge them again. > > We could also always add a CGI script, which Apache can invoke independently > > of Starman. That would have server overheads but wouldn't impact Starman. > > Not an ideal solution though. > > Ultimately I ended up with an in-between container for those reasons.. Could you speak more to that? Is the in-between container another Koha or something else? I think you may have mentioned on Mattermost, but I can't recall what was in that container exactly. Anubis, mod_perl, mod_lua, an in-between container... they're all layers for taking the load off the backend Starman.
Note as per Bug 39206 that if you login and logout of Koha, your KOHA_INIT cookie will be affected.
We are trialling this in Production too and it seems to be working well, thanks David! I want to be really sneaky and propose that we push this fix in time for next week's release, and then improve it in a follow-up. I don't want to shut down the above conversation at all, but bots are the bane of my existence these days and I really like this solution.
I am open to discuss including something in 25.05. I am not sure if we need a way to turn this on/off easily then? We have this week to work something out, I will prepare the release on Monday.
(In reply to David Cook from comment #14) > Note as per Bug 39206 that if you login and logout of Koha, your KOHA_INIT > cookie will be affected. Just add it here to do_not_remove in koha-conf
(In reply to David Cook from comment #13) > Could you speak more to that? Is the in-between container another Koha or > something else? I think you may have mentioned on Mattermost, but I can't > recall what was in that container exactly. Running Starman with a small PSGI script to handle OPAC traffic. Doing cookie check also in Plack middleware level. Either do challenge or pass to Koha rightaway.
(In reply to Katrin Fischer from comment #16) > I am open to discuss including something in 25.05. I am not sure if we need > a way to turn this on/off easily then? We have this week to work something > out, I will prepare the release on Monday. Since it's at the Apache level, I suppose the meaning of "easy" varies from person to person. That said... we could add another RewriteCond that checks if a file exists on the file system. We could then have some mechanism in Koha Admin UI that creates/removes a file in /var/lib/<instance>/something I suppose, although that could be a bit finicky to get right, and I don't know if ${instance} is available at Rewrite time. I'm guessing it would be but haven't tried it myself. If we consider CLI commands to be easy, we could actually move this out of "debian/templates/apache-shared-opac-plack.conf" and use Debian packages to symlink a modified form of "debian/templates/apache-shared-opac-antibot.conf" into /etc/apache2/conf-available/koha-antibot.conf and then require people to use a2enconf/a2disconf to enable/disable it. -- Personally, I'm actually not running this as part of the Koha codebase in prod. I run it at the reverse proxy above Koha (although before doing that I was running it as part of Koha similar to the attached patch).
I think commenting/uncommenting a line in a config file would be easy enough as a first step if we document it well.
(In reply to Katrin Fischer from comment #20) > I think commenting/uncommenting a line in a config file would be easy enough > as a first step if we document it well. That sounds good
We missed release, but this certainly qualifies for a bug fix... ? What is your sucess rate currently?
(In reply to Katrin Fischer from comment #22) > We missed release, but this certainly qualifies for a bug fix... ? What is > your sucess rate currently? I don't know if I could easily quantify it, but it's very good. I have something like this set up on a proxy server, so I see lots of requests coming in to the proxy, and very few get through to the backend Koha server. This config could also be further tuned for library needs.
Created attachment 182906 [details] [review] Bug 39906: Add a simple quick Apache-level bot challenge This patch adds a fast simple bot challenge via Apache. To test: 0. Apply the patch 1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/. 2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/. 3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl 4. Note an instant response with a very simple HTML response 5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl 6. Note a very brief loading animation before the OPAC search page appearing 7. Do a search 8. Note no more loading animation appears 9. Close the incognito browser 10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl 11. Do a search via the "Library catalog" search bar 12. Note no loading animation appears Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de> Signed-off-by: Cornelius Hertfelder <cornelius.hertfelder@bsz-bw.de>
Like to QA this one.
Just a quick update: We've been running these patches in production now on different servers for testing. So far we haven't noticed any problems. But more so: when deploying on a server that was under "attack" we saw the resources used dropping back into a normal range immediately. We are optimistic that for the time being this solution is effective.
We are also really pleased with how this is performing in production.
One QA note... this will likely have a negative impact on cypress/selenium tests, so we'll need to review that before any pushing is done I would say.
(In reply to David Cook from comment #28) > One QA note... this will likely have a negative impact on cypress/selenium > tests, so we'll need to review that before any pushing is done I would say. Maybe we could have it inactive in ktd?
(In reply to Katrin Fischer from comment #29) > (In reply to David Cook from comment #28) > > One QA note... this will likely have a negative impact on cypress/selenium > > tests, so we'll need to review that before any pushing is done I would say. > > Maybe we could have it inactive in ktd? Yes, that sounds like an easy way out. Probably we should not by default enable this feature but make it very easy (change one config line somewhere) to enable it. Note that several will already have a bot solution, and they probably do not want to activate another one.
(In reply to Marcel de Rooy from comment #30) > (In reply to Katrin Fischer from comment #29) > > (In reply to David Cook from comment #28) > > > One QA note... this will likely have a negative impact on cypress/selenium > > > tests, so we'll need to review that before any pushing is done I would say. > > > > Maybe we could have it inactive in ktd? > > Yes, that sounds like an easy way out. Probably we should not by default > enable this feature but make it very easy (change one config line somewhere) > to enable it. Note that several will already have a bot solution, and they > probably do not want to activate another one. Then for the time being we could comment the line in .../templates/apache-shared-opac-plack.conf. Would that do the trick?
(In reply to Katrin Fischer from comment #31) > (In reply to Marcel de Rooy from comment #30) > > (In reply to Katrin Fischer from comment #29) > > > (In reply to David Cook from comment #28) > > > > One QA note... this will likely have a negative impact on cypress/selenium > > > > tests, so we'll need to review that before any pushing is done I would say. > > > > > > Maybe we could have it inactive in ktd? > > > > Yes, that sounds like an easy way out. Probably we should not by default > > enable this feature but make it very easy (change one config line somewhere) > > to enable it. Note that several will already have a bot solution, and they > > probably do not want to activate another one. > > Then for the time being we could comment the line in > .../templates/apache-shared-opac-plack.conf. Would that do the trick? I would turn it around and do it rather in the antibot conf. Will include in qa comment.
QA Comment: Great work. Thanks for sharing. Some comments with varying importance or urgency. [1] Just sending a fake CGISESSID (or a recent anonymous one) or fake KOHA_INIT cookie allows you to bypass. We can make a start with the current setup (much better than nothing). But it feels too simple; we should have something in there that can be validated quickly but hard to guess. (Since we are in Apache here, that might still be a challenge. Or a good reason to move it further away..) [2] Remove the check on CGISESSID? Also in view of point 1. If we harden KOHA_INIT then why stick to CGISESSID which we do not validate here. [3] Rename KOHA_INIT to something with ANTIBOT_whatever ? ANTIBOT_PASS? [4] Default should be off. So I would recommend adding a SetEnvIf line to antibot.conf that sets ANTIBOT_OVERRIDE. Commenting that line and restarting Apache enables the feature. Keeping the comment for further refining on virtual host level. [5] Which scripts should be blocked? +RewriteCond expr "%{REQUEST_URI} =~ m#^/cgi-bin/koha/(opac-detail.pl|opac-export.pl|opac-suggestions.pl|opac-search.pl|opac-authoritiesdetail.pl|opac-ISBDdetail.pl|opac-MARCdetail.pl|opac-shelves.pl)$#" Why not just block every .pl script? And perhaps even some opac/svc stuff (later)? Match on /cgi/bin/koha/ ? Note that I also saw opac-request-article.pl and opac-reserve passing by. opac-retrieve-file is an ugly one too. opac-showmarc etc. In general all .pl scripts are too slow if the bots heavily hit you. [6] No blocker but this would be nice: Add two simple (commented) IP whitelist lines in antibot.conf to bypass the check quickly. Checking either REMOTE_ADDR or HTTP_X_FORWARDED_FOR. This would be just comments for easy local adjustment. So, 1, 2 and 6 may be postponed (6 is easy btw, 1 is not, 2 is linked to 1). 3 and 4 are very easy, please adjust. I am open for discussion on 5, but inclined to require matching /cgi-bin/koha :)
(In reply to David Cook from comment #28) > One QA note... this will likely have a negative impact on cypress/selenium > tests, so we'll need to review that before any pushing is done I would say. When the default is off, that point is handled.
I think I can add thoughts to some of these points, but I believe we might want to keep in mind to get an MVP out fast and them maybe refine further later. > [1] Just sending a fake CGISESSID (or a recent anonymous one) or fake > KOHA_INIT cookie allows you to bypass. We can make a start with the current > setup (much better than nothing). But it feels too simple; we should have > something in there that can be validated quickly but hard to guess. (Since > we are in Apache here, that might still be a challenge. Or a good reason to > move it further away..) > > [2] Remove the check on CGISESSID? Also in view of point 1. If we harden > KOHA_INIT then why stick to CGISESSID which we do not validate here. What we see is that the bots with the varying IP usually don't send any cookies. That explains the immediate drop in load after activating the solution, without invalidating sessions etc. So they have neither the CGISESSID nor the KOHA_INIT cookie. By checking the CGISESSID we don't challenge the Koha users that navigate to the challenged pages from other pages. > [3] Rename KOHA_INIT to something with ANTIBOT_whatever ? ANTIBOT_PASS? I am not sure if it needs to be obvious, but maybe KOHA_INIT is a bit too general? > [4] Default should be off. So I would recommend adding a SetEnvIf line to > antibot.conf that sets ANTIBOT_OVERRIDE. Commenting that line and restarting > Apache enables the feature. Keeping the comment for further refining on > virtual host level. > [5] Which scripts should be blocked? > +RewriteCond expr "%{REQUEST_URI} =~ > m#^/cgi-bin/koha/(opac-detail.pl|opac-export.pl|opac-suggestions.pl|opac- > search.pl|opac-authoritiesdetail.pl|opac-ISBDdetail.pl|opac-MARCdetail. > pl|opac-shelves.pl)$#" > Why not just block every .pl script? And perhaps even some opac/svc stuff > (later)? Match on /cgi/bin/koha/ ? > Note that I also saw opac-request-article.pl and opac-reserve passing by. > opac-retrieve-file is an ugly one too. opac-showmarc etc. In general all .pl > scripts are too slow if the bots heavily hit you. We have discussed this internally at some length when implementing on our servers. From our observations the listed .pl are in fact the pages that get "attacked" the most, especially opac-detail, opac-search and opac-export. More so, these get called "directly" from different IP addresses without navigating the OPAC like a real person would usually do. We believe the URLs are cached, since the attacks continue even after enforcing login for searching (OpacPublic). If we check all pages, we will need to exclude services like ILS-DI, OAI-PMH, REST API... I think having a more careful first iteration with a limited list like this is good. > [6] No blocker but this would be nice: Add two simple (commented) IP > whitelist lines in antibot.conf to bypass the check quickly. Checking either > REMOTE_ADDR or HTTP_X_FORWARDED_FOR. This would be just comments for easy > local adjustment. I like this idea. > So, 1, 2 and 6 may be postponed (6 is easy btw, 1 is not, 2 is linked to 1). > 3 and 4 are very easy, please adjust. I am open for discussion on 5, but > inclined to require matching /cgi-bin/koha :) I think /cgi-bin/koha would be too risky.
I think Katrin answered most of the points. Let me know if there's anything that anyone would like me to specifically clarify. (In reply to Marcel de Rooy from comment #33) > [6] No blocker but this would be nice: Add two simple (commented) IP > whitelist lines in antibot.conf to bypass the check quickly. Checking either > REMOTE_ADDR or HTTP_X_FORWARDED_FOR. This would be just comments for easy > local adjustment. > > So, 1, 2 and 6 may be postponed (6 is easy btw, 1 is not, 2 is linked to 1). Yeah this one is super easy. If your Koha is behind a proxy, I'd add something like this where 192.168.1.100 is the IP address of your reverse proxy: <IfModule mod_remoteip.c> RemoteIPHeader X-Forwarded-For RemoteIPInternalProxy 192.168.1.100/32 </IfModule> #NOTE that you'll need to enable mod_remoteip first and restart Apache Then you can add an exception for a lot of local IPs (if you're using 192.168.1.0/24). RewriteCond %{REMOTE_ADDR} !^192\.168\.1 Remember not to use this last REMOTE_ADDR check without mod_remoteip if you're behind a reverse proxy. Otherwise, everything will be exempt from the bot challenge. -- Personally I left off IP address exemptions from the patch just because IP address restrictions are very subjective/local to individual systems. --
(In reply to Marcel de Rooy from comment #33) > [1] Just sending a fake CGISESSID (or a recent anonymous one) or fake > KOHA_INIT cookie allows you to bypass. We can make a start with the current > setup (much better than nothing). But it feels too simple; we should have > something in there that can be validated quickly but hard to guess. (Since > we are in Apache here, that might still be a challenge. Or a good reason to > move it further away..) > > [2] Remove the check on CGISESSID? Also in view of point 1. If we harden > KOHA_INIT then why stick to CGISESSID which we do not validate here. Katrin already answered these but figured I'd add my 2 cents anyway. As Katrin notes, currently bots don't appear to be sending cookies, so we don't need to validate the contents of the cookies. The existence of the cookie is what we care about at this point. If this changes, it will be challenging, because Apache is limited in what it can do. One option would be to look at something like mod_perl or mod_lua to add logic for that validation, but that would add processing overhead and add complexity to overall Koha installations. Another option would be to use a FastCGI-based authenticator, but same problem. I've got some other ideas for hacky solutions but I don't think I'd want to pursue them. (Note mod_perl requires prefork_mpm which is fine for now but it would be great to move away from prefork_mpm one day.)(I think that mod_lua can be used with event_mpm though.) That said, one simple one could be to set the contents of KOHA_INIT to the IP address of the requestor and then validate that in Apache. That said, people using corporate HTTP forward proxies that have multiple outbound IP addresses could fall victim to failed validations. I haven't tried mod_ext_filter in input mode but that could be interesting. -- But... I figure we cross that bridge when we come to it. No point prematurely optimising I think.
Thx for your replies. About the cookies, we now rely on the bots not sending cookies. So I marked comment1 and comment2 as 'postpone'. But since the cookie contents does not really matter, the bot just needs a very tiny code change to change the game at once. I anticipated discussion on [5] :) REST API was mentioned but it goes via /api, so is not (yet) relevant. oai.pl and ilsdi.pl should be exempted indeed when used, and probably need additional protection measures at some point. I would prefer to match cgi-bin/koha and whitelist a few exceptions (like these two) when those are needed.
(In reply to David Cook from comment #37) > I haven't tried mod_ext_filter in input mode but that could be interesting. "This filtering mechanism is much slower than using a filter which is specially written for the Apache API.."
(In reply to Marcel de Rooy from comment #39) > (In reply to David Cook from comment #37) > > I haven't tried mod_ext_filter in input mode but that could be interesting. > > "This filtering mechanism is much slower than using a filter which is > specially written for the Apache API.." Yep. It's just brainstorming.
(In reply to Marcel de Rooy from comment #38) > About the cookies, we now rely on the bots not sending cookies. So I marked > comment1 and comment2 as 'postpone'. But since the cookie contents does not > really matter, the bot just needs a very tiny code change to change the game > at once. Chances are if they do change, it will be to send real cookies, so that'll change things anyway. I don't see any benefit of trying to speculatively develop for a hypothetical scenario here. > I anticipated discussion on [5] :) REST API was mentioned but it goes via > /api, so is not (yet) relevant. oai.pl and ilsdi.pl should be exempted > indeed when used, and probably need additional protection measures at some > point. > I would prefer to match cgi-bin/koha and whitelist a few exceptions (like > these two) when those are needed. Again it really comes to personal preference here. You even say "I would prefer". There's lots of different scenarios that different organisations could plan for. It's one of the reasons why some people think including this kind of code in Koha is not a good idea in the first place - because it's too subjective. If you do match cgi-bin/koha and allowlist exceptions, it's going to be more work. For instance, your opac-main.pl may disappear or look really weird on Google. Maybe there are other content pages you want to be indexed by Google. So then you make an exception for that. I've provided a list of pages that I've seen hit heavily in the wild. But it's re-configurable by anyone. It's really easy to do for anyone with familiarity with Apache.
So... I have been thinking about an alternative idea to this patch. We don't have to include this in the Koha codebase. We could add a different git repository and set it up as a separate Debian package, which doesn't directly change anything with Koha. We could add a reference to it in the release notes, or have a "Koha in Production" wiki page, or something like that. -- Honestly, I have no intention of using the patch on this bug report, because I'm actually deploying this mechanism in the reverse proxy in front of the Koha server. If we did have a separate Debian package, we could always also tell people to just deploy the "koha-antibot" package on the reverse proxy instead of their Koha server, if that's the setup they have. There's so many potential options here.
(In reply to David Cook from comment #41) > (In reply to Marcel de Rooy from comment #38) > > About the cookies, we now rely on the bots not sending cookies. So I marked > > comment1 and comment2 as 'postpone'. But since the cookie contents does not > > really matter, the bot just needs a very tiny code change to change the game > > at once. > > Chances are if they do change, it will be to send real cookies, so that'll > change things anyway. I don't see any benefit of trying to speculatively > develop for a hypothetical scenario here. Hmm. Speculatively and hypothetical.. I still think that my comment was nuanced enough to not get this kind of terms back.
(In reply to David Cook from comment #41) > Again it really comes to personal preference here. You even say "I would > prefer". There's lots of different scenarios that different organisations > could plan for. It's one of the reasons why some people think including this > kind of code in Koha is not a good idea in the first place - because it's > too subjective. Moving it further away is "my idea" too (as already mentioned). But it may be easier for a lot of libraries to have something in Koha.. Whatever we match, is some choice just as what the patch proposes. > If you do match cgi-bin/koha and allowlist exceptions, it's going to be more > work. For instance, your opac-main.pl may disappear or look really weird on > Google. Maybe there are other content pages you want to be indexed by > Google. So then you make an exception for that. "More work" is subjective too. You could whitelist Google too on a few IP ranges. > I've provided a list of pages that I've seen hit heavily in the wild. But > it's re-configurable by anyone. It's really easy to do for anyone with > familiarity with Apache. Sure. Will put this into In Discussion and leave QA to other people.
(In reply to Marcel de Rooy from comment #43) > Hmm. Speculatively and hypothetical.. I still think that my comment was > nuanced enough to not get this kind of terms back. My apologies. I didn't intend anything negative with those terms. I just mean it would be coding for a scenario that isn't currently happening. That's all.
(In reply to Marcel de Rooy from comment #44) > Moving it further away is "my idea" too (as already mentioned). But it may > be easier for a lot of libraries to have something in Koha.. Whatever we > match, is some choice just as what the patch proposes. When I said "some people", I meant whoever was talking about it on Mattermost. I think that was you, folk from Austria, maybe some other people. > > If you do match cgi-bin/koha and allowlist exceptions, it's going to be more > > work. For instance, your opac-main.pl may disappear or look really weird on > > Google. Maybe there are other content pages you want to be indexed by > > Google. So then you make an exception for that. > "More work" is subjective too. You could whitelist Google too on a few IP > ranges. Well, I'd probably not whitelist Google based on IP, but probably just use their UserAgent string (of course, there have been times where I've wondered if people have been impersonating it), although maybe both would be good... > > I've provided a list of pages that I've seen hit heavily in the wild. But > > it's re-configurable by anyone. It's really easy to do for anyone with > > familiarity with Apache. > Sure. > > Will put this into In Discussion and leave QA to other people. Up to you. If you want to make a patch to include some of your ideas on this bug report, I'd be happy to test/QA it too.
I guess it is time for some new eyes here :)
We're very happy with the antibot effect still. I know a lot of you have other measures in place, but having something simple, low risk like this easily available to the libraries that are currently hitting Mattermost with bot issues, would be great. Especially for the smaller self hosted I believe this would have great value if it was included in Koha's code.
I have tested this patch on one site in production, and it seems to work well, without any negative side effects we are aware of. I agree with Katrin that it would be great to have something like this in Koha, that could easily be activated by those who want/need it, and don't want to investigate other options.
I like the approach, even if it is not perfect. What I see missing is a way to enable/disable it. I can add a koha-anti-bot script if we agree on that. It would use the same approach we use for plack ((un)commenting the include). Ultimately, we will want to change CGISESSID into a JWT that can be validated fast. But that's -as Marcel pointed- way deeper in the stack and will need more thought.
Currently there is an HTML file that pulls in CSS and JS from separate files. Would it be more efficient to have the CSS and JS inline in the HTML file? That would reduce the number of calls to the server from three to one per challenge, unless there is something I am missing.
(In reply to Magnus Enger from comment #51) > Currently there is an HTML file that pulls in CSS and JS from separate > files. Would it be more efficient to have the CSS and JS inline in the HTML > file? That would reduce the number of calls to the server from three to one > per challenge, unless there is something I am missing. The bots generally dont follow these links. They just get the html and do not bother with css and js.
(In reply to Marcel de Rooy from comment #52) > (In reply to Magnus Enger from comment #51) > > Currently there is an HTML file that pulls in CSS and JS from separate > > files. Would it be more efficient to have the CSS and JS inline in the HTML > > file? That would reduce the number of calls to the server from three to one > > per challenge, unless there is something I am missing. > > The bots generally dont follow these links. They just get the html and do > not bother with css and js. Ah, then it makes sense to separate them. Thanks!
(In reply to Tomás Cohen Arazi (tcohen) from comment #50) > I like the approach, even if it is not perfect. > > What I see missing is a way to enable/disable it. I can add a koha-anti-bot > script if we agree on that. It would use the same approach we use for plack > ((un)commenting the include). > > Ultimately, we will want to change CGISESSID into a JWT that can be > validated fast. But that's -as Marcel pointed- way deeper in the stack and > will need more thought. I'd love a koha-script for these as it would make it even simpler for the self-hosted libraries. I'd be happy to test the follow-up.
Created attachment 183590 [details] [review] Bug 39906: [FOR DISCUSSION] Add a worker to waste some CPU cycles if likely bot - The only change to the antibot config is the addition of opac-search.pl (just noticed that the root path should be included as well). - Added a config object to adjust and experiment with the settings. - Added a worker script to calculate the number of primes in a given number of iterations set by index.js.
(In reply to Paul Derscheid from comment #55) > - The only change to the antibot config is the addition of > opac-search.pl (just noticed that the root path should be included as > well). So adding opac-main. Thats fine.
(In reply to Paul Derscheid from comment #55) > - Added a config object to adjust and experiment with the settings. > - Added a worker script to calculate the number of primes in a given > number of iterations set by index.js. Cool. Will have a look
(In reply to Paul Derscheid from comment #55) > - Added a config object to adjust and experiment with the settings. Looks nice > - Added a worker script to calculate the number of primes in a given > number of iterations set by index.js. The only problem sofar is that they are too lazy to run your code ;)
One word of warning for including opac-main.pl. It could mean that your Koha stops being indexed by search engines like Google. For some libraries, that would be a big problem.
(In reply to David Cook from comment #59) > One word of warning for including opac-main.pl. > > It could mean that your Koha stops being indexed by search engines like > Google. For some libraries, that would be a big problem. This is an important point. opac-search.pl and opac-detail.pl are the main 'problematic' pages for us
(In reply to Aleisha Amohia from comment #60) > (In reply to David Cook from comment #59) > > One word of warning for including opac-main.pl. > > > > It could mean that your Koha stops being indexed by search engines like > > Google. For some libraries, that would be a big problem. > > This is an important point. > > opac-search.pl and opac-detail.pl are the main 'problematic' pages for us True indeed. There should be a way to let Googlebot pass by. (Currently I allow a few Google IP ranges.) Checking only the user agent feels unsafe. Note btw that not indexing opac-detail might be an issue as well.
This still works well for us in the initial format. I wonder if some of the points taken could not be solved by a warning message/documentation with the option to refine later and on separate bugs?
(In reply to Katrin Fischer from comment #62) > This still works well for us in the initial format. I wonder if some of the > points taken could not be solved by a warning message/documentation with the > option to refine later and on separate bugs? We support this approach. It this stage we have this live patched because we couldn't keep waiting for the fix to come in a minor point update, and it's been a real saviour for us.
We have my and Cornelius sign-off, we are both from BSZ, so it cannot count for both signoff and QA. If we got more sign-offs I would be willing to put this in PQA (former RM get to QA)
I had offered a signoff for the original patch. I don't know about the FOR DISCUSSION patch - should that be tested?
Created attachment 184784 [details] [review] Bug 39906: Add a simple quick Apache-level bot challenge This patch adds a fast simple bot challenge via Apache. To test: 0. Apply the patch 1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/. 2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/. 3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl 4. Note an instant response with a very simple HTML response 5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl 6. Note a very brief loading animation before the OPAC search page appearing 7. Do a search 8. Note no more loading animation appears 9. Close the incognito browser 10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl 11. Do a search via the "Library catalog" search bar 12. Note no loading animation appears Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de> Signed-off-by: Cornelius Hertfelder <cornelius.hertfelder@bsz-bw.de> Signed-off-by: Magnus Enger <magnus@libriotech.no> We have tested this in production on at least three live sites for several weeks, and it has worked as expected. We have not noticed any negative side effects.
Is there still any feedback about the alternative patch of Paul?
(In reply to Marcel de Rooy from comment #67) > Is there still any feedback about the alternative patch of Paul? Previously, I hadn't tried Paul's Javascript, but I did think that it looked very interesting. It looks like the patch was attached in the wrong order but that's an easy fix at "git bz apply" time... From a UX perspective, it looks similar, although it seems to take 4-5 seconds instead of 1 second. As a human user, I found that off-putting, but I think we could probably tune that. Overall, I think waiting more than 1-2 seconds is probably going to be unpleasant from a human UX perspective. -- I suppose my overall feedback would be... probably needs a bit more testing. Also, personally I'm opposed to including opac-main.pl as I think it could have a negative impact on legitimate web indexing, but that's just my opinion. Maybe it's better to be conservative and include it as a baseline.
(In reply to Marcel de Rooy from comment #58) > The only problem sofar is that they are too lazy to run your code ;) Correct 😅
(In reply to David Cook from comment #69) > (In reply to Marcel de Rooy from comment #58) > > The only problem sofar is that they are too lazy to run your code ;) > > Correct 😅 Not sure if you meant the humans or the bots, but as a human I'll own that one hehe.
Created attachment 184886 [details] [review] Bug 39906: Add a simple quick Apache-level bot challenge This patch adds a fast simple bot challenge via Apache. To test: 0. Apply the patch 1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/. 2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/. 3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl 4. Note an instant response with a very simple HTML response 5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl 6. Note a very brief loading animation before the OPAC search page appearing 7. Do a search 8. Note no more loading animation appears 9. Close the incognito browser 10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl 11. Do a search via the "Library catalog" search bar 12. Note no loading animation appears Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de> Signed-off-by: Cornelius Hertfelder <cornelius.hertfelder@bsz-bw.de> Signed-off-by: Magnus Enger <magnus@libriotech.no> We have tested this in production on at least three live sites for several weeks, and it has worked as expected. We have not noticed any negative side effects. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 184887 [details] [review] Bug 39906: (QA follow-up) Feature needs to be enabled specifically See also comment33, point 4. Test plan: cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/ Test enabling/disabling by toggling the comment of the SetEnvIf line, restart apache and remove the KOHA_INIT and CGISESSID cookie. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Will move the patch from Paul to 40553 for further testing and consideration. Refer the RM to comment33. Note that actually only comment4 has been done (and not by the initial author ;) The feature has been tested well enough now.
Created attachment 184890 [details] [review] Bug 39906: (QA follow-up) Feature needs to be enabled specifically See also comment33, point 4. Test plan: cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/ Test enabling/disabling by toggling the comment of the SetEnvIf line, restart apache and remove the KOHA_INIT and CGISESSID cookie. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Thanks for moving my patch, Marcel. The delay introduced by the additional code is indeed substantial. I experimented with this on a staging instance that was being bombarded with traffic at the time. But doesn't help much if the clients don't even run the JS, that is very true :D
(In reply to Paul Derscheid from comment #75) > The delay introduced by the additional code is indeed substantial. I > experimented with this on a staging instance that was being bombarded with > traffic at the time. Ahh right. That makes sense. Let's talk about it more on bug 40553?
+ requestAnimationFrame(() => { + const delay = performance.now() - timestamp1; + if (delay < 20){ + timeout_incr += 1000; + } David, why 20? It seems that a common refresh rate of 60Hz will lead to values being under 16,7 with an average under 8,35. Could incidentally be 0.
(In reply to Marcel de Rooy from comment #77) > + requestAnimationFrame(() => { > + const delay = performance.now() - timestamp1; > + if (delay < 20){ > + timeout_incr += 1000; > + } > > David, why 20? > It seems that a common refresh rate of 60Hz will lead to values being under > 16,7 with an average under 8,35. Could incidentally be 0. Hmm, I can't remember now. It doesn't really do any harm, but it doesn't do much good either.
Applied to 22.11.x-security for 22.11.30
Applied to 25.05.x-security for 25.05.03.