Bug 39906

Summary: Add bot challenge (in Apache layer)
Product: Koha Reporter: David Cook <dcook>
Component: Architecture, internals, and plumbingAssignee: David Cook <dcook>
Status: Passed QA --- QA Contact: Marcel de Rooy <m.de.rooy>
Severity: enhancement    
Priority: P5 - low CC: aleisha, amitddng135, bibliothek, clemens.tubach, cornelius.amzar, dcook, didier.gautheron, fridolin.somers, jonathan.druart, jonathan.field, katrin.fischer, koha, laurent.ducos, lisette, lucas, lukasz.koszyk, m.de.rooy, magnus, markus.john, martin.renvoize, michael.skarupianski, michaela.sieber, nick, pascal.uphaus, paul.derscheid, phil, philippe.blouin, raphael.straub, tadeusz, tomascohen, wainuiwitikapark
Version: MainKeywords: release-notes-needed
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39435
GIT URL: Change sponsored?: ---
Patch complexity: Small patch Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
25.05.03,24.11.08,24.05.13,22.11.30
Circulation function:
Bug Depends on:    
Bug Blocks: 40553    
Attachments: Bug 39906: Add a simple quick Apache-level bot challenge
Bug 39906: Add a simple quick Apache-level bot challenge
Bug 39906: Add a simple quick Apache-level bot challenge
Bug 39906: Add a simple quick Apache-level bot challenge
Bug 39906: [FOR DISCUSSION] Add a worker to waste some CPU cycles if likely bot
Bug 39906: Add a simple quick Apache-level bot challenge
Bug 39906: Add a simple quick Apache-level bot challenge
Bug 39906: (QA follow-up) Feature needs to be enabled specifically
Bug 39906: (QA follow-up) Feature needs to be enabled specifically

Description David Cook 2025-05-15 06:13:45 UTC
In my experience, Bug 39435 works well, but if you only have a few Starman workers it's still a suboptimal use of resources.

By doing a bot challenge at the Apache reverse proxy level, we're able to save our Starman resources for when we really need them.
Comment 1 David Cook 2025-05-15 06:40:58 UTC
Created attachment 182474 [details] [review]
Bug 39906: Add a simple quick Apache-level bot challenge

This patch adds a fast simple bot challenge via Apache.

To test:
0. Apply the patch
1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/.
2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/.
3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl
4. Note an instant response with a very simple HTML response
5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl
6. Note a very brief loading animation before the OPAC search page appearing
7. Do a search
8. Note no more loading animation appears
9. Close the incognito browser
10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl
11. Do a search via the "Library catalog" search bar
12. Note no loading animation appears
Comment 2 David Cook 2025-05-15 06:41:56 UTC
Note that I don't really think this is "Needs Signoff" but it's in a testable state.

--

One thing to note... this patch will probably cause our Selenium testing to slow down significantly, so there's probably some tweaks to do here.

But this should be a simple enough proof-of-concept.
Comment 3 Marcel de Rooy 2025-05-15 06:56:35 UTC
Nice!
What should be the expiration of KOHA_INIT cookie ?
Comment 4 David Cook 2025-05-15 23:20:31 UTC
(In reply to Marcel de Rooy from comment #3)
> Nice!
> What should be the expiration of KOHA_INIT cookie ?

I think we probably have to keep it as a session expiry, since we can't do a dynamic calculation here.
Comment 5 David Cook 2025-05-16 00:01:33 UTC
(In reply to David Cook from comment #4)
> (In reply to Marcel de Rooy from comment #3)
> > Nice!
> > What should be the expiration of KOHA_INIT cookie ?
> 
> I think we probably have to keep it as a session expiry, since we can't do a
> dynamic calculation here.

Actually, as I was leaving work yesterday, I was thinking about some load testing I was doing and how I wasn't happy with the KOHA_INIT cookie being set in the Apache.

I think it'll be better to move it to the Javascript. I originally was setting the KOHA_INIT cookie when I was sending a 302 in Apache rather than doing an internal redirect. It's not really needed anymore. 

New patch incoming...
Comment 6 David Cook 2025-05-16 00:03:25 UTC
Created attachment 182506 [details] [review]
Bug 39906: Add a simple quick Apache-level bot challenge

This patch adds a fast simple bot challenge via Apache.

To test:
0. Apply the patch
1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/.
2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/.
3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl
4. Note an instant response with a very simple HTML response
5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl
6. Note a very brief loading animation before the OPAC search page appearing
7. Do a search
8. Note no more loading animation appears
9. Close the incognito browser
10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl
11. Do a search via the "Library catalog" search bar
12. Note no loading animation appears
Comment 7 David Cook 2025-05-16 00:20:37 UTC
Ok now we get into the fun part...

To load test:
0. apt-get install siege
1. siege -d 1 -c 12 -t 300s --no-parser --internet "http://127.0.0.1:8080/cgi-bin/koha/opac-search.pl"
1b. Tweak the concurrent request -c parameter as necessary. If you have a very low spec machine, this may not be a good test (as your machine will be handles you, siege, KTD, etc)
2. Try navigating your KTD OPAC and staff interface
3. Note that it still feels usable (depending on the number of CPUs you have available to KTD of course)
4. sudo tail -f /var/log/apache2/other_vhosts_access.log
5. Note the huge flood of incoming requests from Siege being met with small responses back
6. ss -t -n -l
7. Note that the Recv-Q for 0.0.0.0:8080 is 0 or low (again depends on the number of CPUs you have available
Comment 8 David Cook 2025-05-16 02:01:25 UTC
Created attachment 182507 [details] [review]
Bug 39906: Add a simple quick Apache-level bot challenge

This patch adds a fast simple bot challenge via Apache.

To test:
0. Apply the patch
1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/.
2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/.
3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl
4. Note an instant response with a very simple HTML response
5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl
6. Note a very brief loading animation before the OPAC search page appearing
7. Do a search
8. Note no more loading animation appears
9. Close the incognito browser
10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl
11. Do a search via the "Library catalog" search bar
12. Note no loading animation appears
Comment 9 David Cook 2025-05-16 02:02:40 UTC
Fixed up the HTTPS check to add "Secure" to the cookie. Not really necessary I guess, but I figure why not.
Comment 10 Katrin Fischer 2025-05-19 13:11:08 UTC
We can confirm that the solution works as described on a Koha 22.11.26 instance. We cannot confirm yet how effective the solution it, but maybe soon.

Some questions that popped up looking at the solution:

* Wondering if the cookie value could be validated in some way. Currently we have KOHA_INIT=1 in the cookie. Could 1 be switched to a hash value for example or something else cryptic that is ideally also verified by the server?

* We have monitoring that checks if the search is still working. That will also be affected by the challenge on opac-search. Would it be possible to allow certain IPs to get to the page unchallenged?
Comment 11 David Cook 2025-05-19 23:40:15 UTC
(In reply to Katrin Fischer from comment #10)
> Some questions that popped up looking at the solution:
> 
> * Wondering if the cookie value could be validated in some way. Currently we
> have KOHA_INIT=1 in the cookie. Could 1 be switched to a hash value for
> example or something else cryptic that is ideally also verified by the
> server?

Right now, the existence of the KOHA_INIT cookie is all that Apache checks (the value is irrelevant), and Apache is quite limited in what it can do without the use of something like mod_perl or mod_lua.

But yeah... there's things that could be done. Rather than checking for the presence of KOHA_INIT in Apache, you could always pass through to the Javascript, and have it do a check.

But... then if the check involves an API call that's going to be additional load on the Koha server. It might seem small but when you start scaling up the requests, that's when Starman starts to suffer. 

In practice, I haven't found a need to validate the contents of the KOHA_INIT cookie, but in the future it might be necessary. 

An easy one to do could be to actually store the client's IP address in KOHA_INIT. That way, if the KOHA_INIT is sent by a different IP address than the one that generated the cookie, then it would be an easy catch/validation. 

Other than that... someone could experiment with Apache but they might have to go the mod_perl or mod_lua route. mod_perl would probably make sense for Koha people since we're already familiar with Perl.

I use mod_perl a lot in a different non-Koha app, so I can probably help out with that. 

I suppose one thing that could be experimented with is mod_expr. 

We could also always add a CGI script, which Apache can invoke independently of Starman. That would have server overheads but wouldn't impact Starman. Not an ideal solution though.

On a similar note, we could do a FCGI handler, but then that seems like overkill. 

Overall, I'd say there are options, but... they're not really necessary at the moment. 
 
> * We have monitoring that checks if the search is still working. That will
> also be affected by the challenge on opac-search. Would it be possible to
> allow certain IPs to get to the page unchallenged?

Yes, absolutely. That's what I do locally. All you need to do is add a RewriteCond in that debian/templates/apache-shared-opac-antibot.conf file. 

For example, if you want to allow everything on 192.168.1.0/24, you'd just add:
RewriteCond %{REMOTE_ADDR} !^192\.168\.1

That way, only IP addresses other than that one would trigger the condition.
Comment 12 Marcel de Rooy 2025-05-20 11:24:08 UTC
(In reply to David Cook from comment #11)

> An easy one to do could be to actually store the client's IP address in
> KOHA_INIT. That way, if the KOHA_INIT is sent by a different IP address than
> the one that generated the cookie, then it would be an easy
> catch/validation. 

Remenber that people may change IP while travelling. So you need to decide if the cookie is IP independent or another IP needs a new cookie (probably better: if one bot obtains the cookie, do not allow them all).

> We could also always add a CGI script, which Apache can invoke independently
> of Starman. That would have server overheads but wouldn't impact Starman.
> Not an ideal solution though.

Ultimately I ended up with an in-between container for those reasons..
Comment 13 David Cook 2025-05-20 23:23:32 UTC
(In reply to Marcel de Rooy from comment #12)
> Remenber that people may change IP while travelling. So you need to decide
> if the cookie is IP independent or another IP needs a new cookie (probably
> better: if one bot obtains the cookie, do not allow them all).

Yeah, I've thought about that scenario as well. Or they might be using a VPN or HTTP forward proxy that has a range of outbound IP addresses. 

However, since is just needed for a top-level navigation, it doesn't really matter if we challenge them again. 
 
> > We could also always add a CGI script, which Apache can invoke independently
> > of Starman. That would have server overheads but wouldn't impact Starman.
> > Not an ideal solution though.
> 
> Ultimately I ended up with an in-between container for those reasons..

Could you speak more to that? Is the in-between container another Koha or something else? I think you may have mentioned on Mattermost, but I can't recall what was in that container exactly. 

Anubis, mod_perl, mod_lua, an in-between container... they're all layers for taking the load off the backend Starman.
Comment 14 David Cook 2025-05-20 23:25:36 UTC
Note as per Bug 39206 that if you login and logout of Koha, your KOHA_INIT cookie will be affected.
Comment 15 Aleisha Amohia 2025-05-21 02:22:35 UTC
We are trialling this in Production too and it seems to be working well, thanks David!

I want to be really sneaky and propose that we push this fix in time for next week's release, and then improve it in a follow-up. I don't want to shut down the above conversation at all, but bots are the bane of my existence these days and I really like this solution.
Comment 16 Katrin Fischer 2025-05-21 08:47:06 UTC
I am open to discuss including something in 25.05. I am not sure if we need a way to turn this on/off easily then? We have this week to work something out, I will prepare the release on Monday.
Comment 17 Marcel de Rooy 2025-05-21 11:10:03 UTC
(In reply to David Cook from comment #14)
> Note as per Bug 39206 that if you login and logout of Koha, your KOHA_INIT
> cookie will be affected.

Just add it here to do_not_remove in koha-conf
Comment 18 Marcel de Rooy 2025-05-21 11:15:10 UTC
(In reply to David Cook from comment #13)
> Could you speak more to that? Is the in-between container another Koha or
> something else? I think you may have mentioned on Mattermost, but I can't
> recall what was in that container exactly. 

Running Starman with a small PSGI script to handle OPAC traffic. Doing cookie check also in Plack middleware level. Either do challenge or pass to Koha rightaway.
Comment 19 David Cook 2025-05-21 23:25:47 UTC
(In reply to Katrin Fischer from comment #16)
> I am open to discuss including something in 25.05. I am not sure if we need
> a way to turn this on/off easily then? We have this week to work something
> out, I will prepare the release on Monday.

Since it's at the Apache level, I suppose the meaning of "easy" varies from person to person. 

That said... we could add another RewriteCond that checks if a file exists on the file system. We could then have some mechanism in Koha Admin UI that creates/removes a file in /var/lib/<instance>/something I suppose, although that could be a bit finicky to get right, and I don't know if ${instance} is available at Rewrite time. I'm guessing it would be but haven't tried it myself.

If we consider CLI commands to be easy, we could actually move this out of "debian/templates/apache-shared-opac-plack.conf" and use Debian packages to symlink a modified form of "debian/templates/apache-shared-opac-antibot.conf" into /etc/apache2/conf-available/koha-antibot.conf and then require people to use a2enconf/a2disconf to enable/disable it.

--

Personally, I'm actually not running this as part of the Koha codebase in prod. I run it at the reverse proxy above Koha (although before doing that I was running it as part of Koha similar to the attached patch).
Comment 20 Katrin Fischer 2025-05-23 07:16:18 UTC
I think commenting/uncommenting a line in a config file would be easy enough as a first step if we document it well.
Comment 21 Aleisha Amohia 2025-05-26 22:54:02 UTC
(In reply to Katrin Fischer from comment #20)
> I think commenting/uncommenting a line in a config file would be easy enough
> as a first step if we document it well.

That sounds good
Comment 22 Katrin Fischer 2025-05-30 08:12:47 UTC
We missed release, but this certainly qualifies for a bug fix... ? What is your sucess rate currently?
Comment 23 David Cook 2025-06-02 00:10:59 UTC
(In reply to Katrin Fischer from comment #22)
> We missed release, but this certainly qualifies for a bug fix... ? What is
> your sucess rate currently?

I don't know if I could easily quantify it, but it's very good. I have something like this set up on a proxy server, so I see lots of requests coming in to the proxy, and very few get through to the backend Koha server. 

This config could also be further tuned for library needs.
Comment 24 Katrin Fischer 2025-06-02 11:59:40 UTC
Created attachment 182906 [details] [review]
Bug 39906: Add a simple quick Apache-level bot challenge

This patch adds a fast simple bot challenge via Apache.

To test:
0. Apply the patch
1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/.
2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/.
3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl
4. Note an instant response with a very simple HTML response
5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl
6. Note a very brief loading animation before the OPAC search page appearing
7. Do a search
8. Note no more loading animation appears
9. Close the incognito browser
10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl
11. Do a search via the "Library catalog" search bar
12. Note no loading animation appears

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Cornelius Hertfelder <cornelius.hertfelder@bsz-bw.de>
Comment 25 Marcel de Rooy 2025-06-02 12:06:18 UTC
Like to QA this one.
Comment 26 Katrin Fischer 2025-06-02 12:15:47 UTC
Just a quick update:

We've been running these patches in production now on different servers for testing. So far we haven't noticed any problems. But more so: when deploying on a server that was under "attack" we saw the resources used dropping back into a normal range immediately. We are optimistic that for the time being this solution is effective.
Comment 27 Aleisha Amohia 2025-06-02 21:09:56 UTC
We are also really pleased with how this is performing in production.
Comment 28 David Cook 2025-06-02 23:17:34 UTC
One QA note... this will likely have a negative impact on cypress/selenium tests, so we'll need to review that before any pushing is done I would say.
Comment 29 Katrin Fischer 2025-06-03 08:52:19 UTC
(In reply to David Cook from comment #28)
> One QA note... this will likely have a negative impact on cypress/selenium
> tests, so we'll need to review that before any pushing is done I would say.

Maybe we could have it inactive in ktd?
Comment 30 Marcel de Rooy 2025-06-03 11:12:20 UTC
(In reply to Katrin Fischer from comment #29)
> (In reply to David Cook from comment #28)
> > One QA note... this will likely have a negative impact on cypress/selenium
> > tests, so we'll need to review that before any pushing is done I would say.
> 
> Maybe we could have it inactive in ktd?

Yes, that sounds like an easy way out. Probably we should not by default enable this feature but make it very easy (change one config line somewhere) to enable it. Note that several will already have a bot solution, and they probably do not want to activate another one.
Comment 31 Katrin Fischer 2025-06-03 12:07:42 UTC
(In reply to Marcel de Rooy from comment #30)
> (In reply to Katrin Fischer from comment #29)
> > (In reply to David Cook from comment #28)
> > > One QA note... this will likely have a negative impact on cypress/selenium
> > > tests, so we'll need to review that before any pushing is done I would say.
> > 
> > Maybe we could have it inactive in ktd?
> 
> Yes, that sounds like an easy way out. Probably we should not by default
> enable this feature but make it very easy (change one config line somewhere)
> to enable it. Note that several will already have a bot solution, and they
> probably do not want to activate another one.

Then for the time being we could comment the line in  .../templates/apache-shared-opac-plack.conf. Would that do the trick?
Comment 32 Marcel de Rooy 2025-06-03 14:25:48 UTC
(In reply to Katrin Fischer from comment #31)
> (In reply to Marcel de Rooy from comment #30)
> > (In reply to Katrin Fischer from comment #29)
> > > (In reply to David Cook from comment #28)
> > > > One QA note... this will likely have a negative impact on cypress/selenium
> > > > tests, so we'll need to review that before any pushing is done I would say.
> > > 
> > > Maybe we could have it inactive in ktd?
> > 
> > Yes, that sounds like an easy way out. Probably we should not by default
> > enable this feature but make it very easy (change one config line somewhere)
> > to enable it. Note that several will already have a bot solution, and they
> > probably do not want to activate another one.
> 
> Then for the time being we could comment the line in 
> .../templates/apache-shared-opac-plack.conf. Would that do the trick?

I would turn it around and do it rather in the antibot conf. Will include in qa comment.
Comment 33 Marcel de Rooy 2025-06-03 14:34:35 UTC
QA Comment:
Great work. Thanks for sharing. Some comments with varying importance or urgency.

[1] Just sending a fake CGISESSID (or a recent anonymous one) or fake KOHA_INIT cookie allows you to bypass. We can make a start with the current setup (much better than nothing). But it feels too simple; we should have something in there that can be validated quickly but hard to guess. (Since we are in Apache here, that might still be a challenge. Or a good reason to move it further away..)

[2] Remove the check on CGISESSID? Also in view of point 1. If we harden KOHA_INIT then why stick to CGISESSID which we do not validate here.

[3] Rename KOHA_INIT to something with ANTIBOT_whatever ? ANTIBOT_PASS?

[4] Default should be off. So I would recommend adding a SetEnvIf line to antibot.conf that sets ANTIBOT_OVERRIDE. Commenting that line and restarting Apache enables the feature. Keeping the comment for further refining on virtual host level.

[5] Which scripts should be blocked?
+RewriteCond expr "%{REQUEST_URI} =~ m#^/cgi-bin/koha/(opac-detail.pl|opac-export.pl|opac-suggestions.pl|opac-search.pl|opac-authoritiesdetail.pl|opac-ISBDdetail.pl|opac-MARCdetail.pl|opac-shelves.pl)$#"
Why not just block every .pl script? And perhaps even some opac/svc stuff (later)? Match on /cgi/bin/koha/ ?
Note that I also saw opac-request-article.pl and opac-reserve passing by. opac-retrieve-file is an ugly one too. opac-showmarc etc. In general all .pl scripts are too slow if the bots heavily hit you.

[6] No blocker but this would be nice: Add two simple (commented) IP whitelist lines in antibot.conf to bypass the check quickly. Checking either REMOTE_ADDR or HTTP_X_FORWARDED_FOR. This would be just comments for easy local adjustment.

So, 1, 2 and 6 may be postponed (6 is easy btw, 1 is not, 2 is linked to 1). 3 and 4 are very easy, please adjust. I am open for discussion on 5, but inclined to require matching /cgi-bin/koha :)
Comment 34 Marcel de Rooy 2025-06-03 14:36:33 UTC
(In reply to David Cook from comment #28)
> One QA note... this will likely have a negative impact on cypress/selenium
> tests, so we'll need to review that before any pushing is done I would say.

When the default is off, that point is handled.
Comment 35 Katrin Fischer 2025-06-03 16:11:00 UTC
I think I can add thoughts to some of these points, but I believe we might want to keep in mind to get an MVP out fast and them maybe refine further later.

> [1] Just sending a fake CGISESSID (or a recent anonymous one) or fake
> KOHA_INIT cookie allows you to bypass. We can make a start with the current
> setup (much better than nothing). But it feels too simple; we should have
> something in there that can be validated quickly but hard to guess. (Since
> we are in Apache here, that might still be a challenge. Or a good reason to
> move it further away..)
> 
> [2] Remove the check on CGISESSID? Also in view of point 1. If we harden
> KOHA_INIT then why stick to CGISESSID which we do not validate here.

What we see is that the bots with the varying IP usually don't send any cookies. That explains the immediate drop in load after activating the solution, without invalidating sessions etc. So they have neither the CGISESSID nor the KOHA_INIT cookie. 

By checking the CGISESSID we don't challenge the Koha users that navigate to the challenged pages from other pages.


> [3] Rename KOHA_INIT to something with ANTIBOT_whatever ? ANTIBOT_PASS?

I am not sure if it needs to be obvious, but maybe KOHA_INIT is a bit too general?

> [4] Default should be off. So I would recommend adding a SetEnvIf line to
> antibot.conf that sets ANTIBOT_OVERRIDE. Commenting that line and restarting
> Apache enables the feature. Keeping the comment for further refining on
> virtual host level.

> [5] Which scripts should be blocked?
> +RewriteCond expr "%{REQUEST_URI} =~
> m#^/cgi-bin/koha/(opac-detail.pl|opac-export.pl|opac-suggestions.pl|opac-
> search.pl|opac-authoritiesdetail.pl|opac-ISBDdetail.pl|opac-MARCdetail.
> pl|opac-shelves.pl)$#"
> Why not just block every .pl script? And perhaps even some opac/svc stuff
> (later)? Match on /cgi/bin/koha/ ?
> Note that I also saw opac-request-article.pl and opac-reserve passing by.
> opac-retrieve-file is an ugly one too. opac-showmarc etc. In general all .pl
> scripts are too slow if the bots heavily hit you.

We have discussed this internally at some length when implementing on our servers.

From our observations the listed .pl are in fact the pages that get "attacked" the most, especially opac-detail, opac-search and opac-export. More so, these get called "directly" from different IP addresses without navigating the OPAC like a real person would usually do. We believe the URLs are cached, since the attacks continue even after enforcing login for searching (OpacPublic). If we check all pages, we will need to exclude services like ILS-DI, OAI-PMH, REST API... I think having a more careful first iteration with a limited list like this is good. 

> [6] No blocker but this would be nice: Add two simple (commented) IP
> whitelist lines in antibot.conf to bypass the check quickly. Checking either
> REMOTE_ADDR or HTTP_X_FORWARDED_FOR. This would be just comments for easy
> local adjustment.

I like this idea.

> So, 1, 2 and 6 may be postponed (6 is easy btw, 1 is not, 2 is linked to 1).
> 3 and 4 are very easy, please adjust. I am open for discussion on 5, but
> inclined to require matching /cgi-bin/koha :)

I think /cgi-bin/koha would be too risky.
Comment 36 David Cook 2025-06-04 00:24:01 UTC
I think Katrin answered most of the points. Let me know if there's anything that anyone would like me to specifically clarify.

(In reply to Marcel de Rooy from comment #33)
> [6] No blocker but this would be nice: Add two simple (commented) IP
> whitelist lines in antibot.conf to bypass the check quickly. Checking either
> REMOTE_ADDR or HTTP_X_FORWARDED_FOR. This would be just comments for easy
> local adjustment.
> 
> So, 1, 2 and 6 may be postponed (6 is easy btw, 1 is not, 2 is linked to 1).

Yeah this one is super easy.

If your Koha is behind a proxy, I'd add something like this where 192.168.1.100 is the IP address of your reverse proxy:
<IfModule mod_remoteip.c>
    RemoteIPHeader X-Forwarded-For
    RemoteIPInternalProxy 192.168.1.100/32
</IfModule>
#NOTE that you'll need to enable mod_remoteip first and restart Apache

Then you can add an exception for a lot of local IPs (if you're using 192.168.1.0/24).
RewriteCond %{REMOTE_ADDR} !^192\.168\.1

Remember not to use this last REMOTE_ADDR check without mod_remoteip if you're behind a reverse proxy. Otherwise, everything will be exempt from the bot challenge.

--

Personally I left off IP address exemptions from the patch just because IP address restrictions are very subjective/local to individual systems.
--
Comment 37 David Cook 2025-06-04 00:41:19 UTC
(In reply to Marcel de Rooy from comment #33)
> [1] Just sending a fake CGISESSID (or a recent anonymous one) or fake
> KOHA_INIT cookie allows you to bypass. We can make a start with the current
> setup (much better than nothing). But it feels too simple; we should have
> something in there that can be validated quickly but hard to guess. (Since
> we are in Apache here, that might still be a challenge. Or a good reason to
> move it further away..)
> 
> [2] Remove the check on CGISESSID? Also in view of point 1. If we harden
> KOHA_INIT then why stick to CGISESSID which we do not validate here.

Katrin already answered these but figured I'd add my 2 cents anyway.

As Katrin notes, currently bots don't appear to be sending cookies, so we don't need to validate the contents of the cookies. The existence of the cookie is what we care about at this point.

If this changes, it will be challenging, because Apache is limited in what it can do.  One option would be to look at something like mod_perl or mod_lua to add logic for that validation, but that would add processing overhead and add complexity to overall Koha installations. Another option would be to use a FastCGI-based authenticator, but same problem. I've got some other ideas for hacky solutions but I don't think I'd want to pursue them. (Note mod_perl requires prefork_mpm which is fine for now but it would be great to move away from prefork_mpm one day.)(I think that mod_lua can be used with event_mpm though.)

That said, one simple one could be to set the contents of KOHA_INIT to the IP address of the requestor and then validate that in Apache. That said, people using corporate HTTP forward proxies that have multiple outbound IP addresses could fall victim to failed validations.

I haven't tried mod_ext_filter in input mode but that could be interesting. 

--

But... I figure we cross that bridge when we come to it. No point prematurely optimising I think.
Comment 38 Marcel de Rooy 2025-06-04 11:51:25 UTC
Thx for your replies.

About the cookies, we now rely on the bots not sending cookies. So I marked comment1 and comment2 as 'postpone'. But since the cookie contents does not really matter, the bot just needs a very tiny code change to change the game at once.

I anticipated discussion on [5] :) REST API was mentioned but it goes via /api, so is not (yet) relevant. oai.pl and ilsdi.pl should be exempted indeed when used, and probably need additional protection measures at some point.
I would prefer to match cgi-bin/koha and whitelist a few exceptions (like these two) when those are needed.
Comment 39 Marcel de Rooy 2025-06-04 12:01:25 UTC
(In reply to David Cook from comment #37)
> I haven't tried mod_ext_filter in input mode but that could be interesting. 

"This filtering mechanism is much slower than using a filter which is specially written for the Apache API.."
Comment 40 David Cook 2025-06-05 01:31:50 UTC
(In reply to Marcel de Rooy from comment #39)
> (In reply to David Cook from comment #37)
> > I haven't tried mod_ext_filter in input mode but that could be interesting. 
> 
> "This filtering mechanism is much slower than using a filter which is
> specially written for the Apache API.."

Yep. It's just brainstorming.
Comment 41 David Cook 2025-06-05 01:37:13 UTC
(In reply to Marcel de Rooy from comment #38)
> About the cookies, we now rely on the bots not sending cookies. So I marked
> comment1 and comment2 as 'postpone'. But since the cookie contents does not
> really matter, the bot just needs a very tiny code change to change the game
> at once.

Chances are if they do change, it will be to send real cookies, so that'll change things anyway.  I don't see any benefit of trying to speculatively develop for a hypothetical scenario here.
 
> I anticipated discussion on [5] :) REST API was mentioned but it goes via
> /api, so is not (yet) relevant. oai.pl and ilsdi.pl should be exempted
> indeed when used, and probably need additional protection measures at some
> point.
> I would prefer to match cgi-bin/koha and whitelist a few exceptions (like
> these two) when those are needed.

Again it really comes to personal preference here. You even say "I would prefer". There's lots of different scenarios that different organisations could plan for. It's one of the reasons why some people think including this kind of code in Koha is not a good idea in the first place - because it's too subjective. 

If you do match cgi-bin/koha and allowlist exceptions, it's going to be more work. For instance, your opac-main.pl may disappear or look really weird on Google. Maybe there are other content pages you want to be indexed by Google. So then you make an exception for that. 

I've provided a list of pages that I've seen hit heavily in the wild. But it's re-configurable by anyone. It's really easy to do for anyone with familiarity with Apache.
Comment 42 David Cook 2025-06-05 01:40:24 UTC
So... I have been thinking about an alternative idea to this patch.

We don't have to include this in the Koha codebase. We could add a different git repository and set it up as a separate Debian package, which doesn't directly change anything with Koha.

We could add a reference to it in the release notes, or have a "Koha in Production" wiki page, or something like that.

--

Honestly, I have no intention of using the patch on this bug report, because I'm actually deploying this mechanism in the reverse proxy in front of the Koha server.

If we did have a separate Debian package, we could always also tell people to just deploy the "koha-antibot" package on the reverse proxy instead of their Koha server, if that's the setup they have. 

There's so many potential options here.
Comment 43 Marcel de Rooy 2025-06-05 08:09:49 UTC
(In reply to David Cook from comment #41)
> (In reply to Marcel de Rooy from comment #38)
> > About the cookies, we now rely on the bots not sending cookies. So I marked
> > comment1 and comment2 as 'postpone'. But since the cookie contents does not
> > really matter, the bot just needs a very tiny code change to change the game
> > at once.
> 
> Chances are if they do change, it will be to send real cookies, so that'll
> change things anyway.  I don't see any benefit of trying to speculatively
> develop for a hypothetical scenario here.

Hmm. Speculatively and hypothetical.. I still think that my comment was nuanced enough to not get this kind of terms back.
Comment 44 Marcel de Rooy 2025-06-05 08:15:29 UTC
(In reply to David Cook from comment #41)
> Again it really comes to personal preference here. You even say "I would
> prefer". There's lots of different scenarios that different organisations
> could plan for. It's one of the reasons why some people think including this
> kind of code in Koha is not a good idea in the first place - because it's
> too subjective.
Moving it further away is "my idea" too (as already mentioned). But it may be easier for a lot of libraries to have something in Koha.. Whatever we match, is some choice just as what the patch proposes.

> If you do match cgi-bin/koha and allowlist exceptions, it's going to be more
> work. For instance, your opac-main.pl may disappear or look really weird on
> Google. Maybe there are other content pages you want to be indexed by
> Google. So then you make an exception for that.
"More work" is subjective too. You could whitelist Google too on a few IP ranges.


> I've provided a list of pages that I've seen hit heavily in the wild. But
> it's re-configurable by anyone. It's really easy to do for anyone with
> familiarity with Apache.
Sure.

Will put this into In Discussion and leave QA to other people.
Comment 45 David Cook 2025-06-06 02:43:16 UTC
(In reply to Marcel de Rooy from comment #43)
> Hmm. Speculatively and hypothetical.. I still think that my comment was
> nuanced enough to not get this kind of terms back.

My apologies. I didn't intend anything negative with those terms. I just mean it would be coding for a scenario that isn't currently happening. That's all.
Comment 46 David Cook 2025-06-06 02:47:27 UTC
(In reply to Marcel de Rooy from comment #44)
> Moving it further away is "my idea" too (as already mentioned). But it may
> be easier for a lot of libraries to have something in Koha.. Whatever we
> match, is some choice just as what the patch proposes.

When I said "some people", I meant whoever was talking about it on Mattermost. I think that was you, folk from Austria, maybe some other people. 

> > If you do match cgi-bin/koha and allowlist exceptions, it's going to be more
> > work. For instance, your opac-main.pl may disappear or look really weird on
> > Google. Maybe there are other content pages you want to be indexed by
> > Google. So then you make an exception for that.
> "More work" is subjective too. You could whitelist Google too on a few IP
> ranges.

Well, I'd probably not whitelist Google based on IP, but probably just use their UserAgent string (of course, there have been times where I've wondered if people have been impersonating it), although maybe both would be good...

> > I've provided a list of pages that I've seen hit heavily in the wild. But
> > it's re-configurable by anyone. It's really easy to do for anyone with
> > familiarity with Apache.
> Sure.
> 
> Will put this into In Discussion and leave QA to other people.

Up to you. If you want to make a patch to include some of your ideas on this bug report, I'd be happy to test/QA it too.
Comment 47 Marcel de Rooy 2025-06-06 06:10:46 UTC
I guess it is time for some new eyes here :)
Comment 48 Katrin Fischer 2025-06-10 09:33:33 UTC
We're very happy with the antibot effect still.

I know a lot of you have other measures in place, but having something simple, low risk like this easily available to the libraries that are currently hitting Mattermost with bot issues, would be great. Especially for the smaller self hosted I believe this would have great value if it was included in Koha's code.
Comment 49 Magnus Enger 2025-06-10 09:53:39 UTC
I have tested this patch on one site in production, and it seems to work well, without any negative side effects we are aware of. I agree with Katrin that it would be great to have something like this in Koha, that could easily be activated by those who want/need it, and don't want to investigate other options.
Comment 50 Tomás Cohen Arazi (tcohen) 2025-06-11 17:04:59 UTC
I like the approach, even if it is not perfect.

What I see missing is a way to enable/disable it. I can add a koha-anti-bot script if we agree on that. It would use the same approach we use for plack ((un)commenting the include).

Ultimately, we will want to change CGISESSID into a JWT that can be validated fast. But that's -as Marcel pointed- way deeper in the stack and will need more thought.
Comment 51 Magnus Enger 2025-06-13 07:29:05 UTC
Currently there is an HTML file that pulls in CSS and JS from separate files. Would it be more efficient to have the CSS and JS inline in the HTML file? That would reduce the number of calls to the server from three to one per challenge, unless there is something I am missing.
Comment 52 Marcel de Rooy 2025-06-13 07:55:43 UTC
(In reply to Magnus Enger from comment #51)
> Currently there is an HTML file that pulls in CSS and JS from separate
> files. Would it be more efficient to have the CSS and JS inline in the HTML
> file? That would reduce the number of calls to the server from three to one
> per challenge, unless there is something I am missing.

The bots generally dont follow these links. They just get the html and do not bother with css and js.
Comment 53 Magnus Enger 2025-06-13 08:37:14 UTC
(In reply to Marcel de Rooy from comment #52)
> (In reply to Magnus Enger from comment #51)
> > Currently there is an HTML file that pulls in CSS and JS from separate
> > files. Would it be more efficient to have the CSS and JS inline in the HTML
> > file? That would reduce the number of calls to the server from three to one
> > per challenge, unless there is something I am missing.
> 
> The bots generally dont follow these links. They just get the html and do
> not bother with css and js.

Ah, then it makes sense to separate them. Thanks!
Comment 54 Katrin Fischer 2025-06-16 16:17:33 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #50)
> I like the approach, even if it is not perfect.
> 
> What I see missing is a way to enable/disable it. I can add a koha-anti-bot
> script if we agree on that. It would use the same approach we use for plack
> ((un)commenting the include).
> 
> Ultimately, we will want to change CGISESSID into a JWT that can be
> validated fast. But that's -as Marcel pointed- way deeper in the stack and
> will need more thought.

I'd love a koha-script for these as it would make it even simpler for the self-hosted libraries. I'd be happy to test the follow-up.
Comment 55 Paul Derscheid 2025-06-27 08:14:02 UTC
Created attachment 183590 [details] [review]
Bug 39906: [FOR DISCUSSION] Add a worker to waste some CPU cycles if likely bot

- The only change to the antibot config is the addition of
  opac-search.pl (just noticed that the root path should be included as
  well).
- Added a config object to adjust and experiment with the settings.
- Added a worker script to calculate the number of primes in a given
  number of iterations set by index.js.
Comment 56 Marcel de Rooy 2025-06-27 08:20:50 UTC
(In reply to Paul Derscheid from comment #55)
> - The only change to the antibot config is the addition of
>   opac-search.pl (just noticed that the root path should be included as
>   well).
So adding opac-main. Thats fine.
Comment 57 Marcel de Rooy 2025-06-27 08:21:33 UTC
(In reply to Paul Derscheid from comment #55)
> - Added a config object to adjust and experiment with the settings.
> - Added a worker script to calculate the number of primes in a given
>   number of iterations set by index.js.

Cool. Will have a look
Comment 58 Marcel de Rooy 2025-06-27 10:01:36 UTC
(In reply to Paul Derscheid from comment #55)
> - Added a config object to adjust and experiment with the settings.
Looks nice
> - Added a worker script to calculate the number of primes in a given
>   number of iterations set by index.js.
The only problem sofar is that they are too lazy to run your code ;)
Comment 59 David Cook 2025-06-29 23:33:33 UTC
One word of warning for including opac-main.pl. 

It could mean that your Koha stops being indexed by search engines like Google. For some libraries, that would be a big problem.
Comment 60 Aleisha Amohia 2025-06-29 23:44:25 UTC
(In reply to David Cook from comment #59)
> One word of warning for including opac-main.pl. 
> 
> It could mean that your Koha stops being indexed by search engines like
> Google. For some libraries, that would be a big problem.

This is an important point.

opac-search.pl and opac-detail.pl are the main 'problematic' pages for us
Comment 61 Marcel de Rooy 2025-06-30 06:39:24 UTC
(In reply to Aleisha Amohia from comment #60)
> (In reply to David Cook from comment #59)
> > One word of warning for including opac-main.pl. 
> > 
> > It could mean that your Koha stops being indexed by search engines like
> > Google. For some libraries, that would be a big problem.
> 
> This is an important point.
> 
> opac-search.pl and opac-detail.pl are the main 'problematic' pages for us

True indeed. There should be a way to let Googlebot pass by. (Currently I allow a few Google IP ranges.) Checking only the user agent feels unsafe.

Note btw that not indexing opac-detail might be an issue as well.
Comment 62 Katrin Fischer 2025-07-09 15:56:23 UTC
This still works well for us in the initial format. I wonder if some of the points taken could not be solved by a warning message/documentation with the option to refine later and on separate bugs?
Comment 63 Aleisha Amohia 2025-07-09 21:18:37 UTC
(In reply to Katrin Fischer from comment #62)
> This still works well for us in the initial format. I wonder if some of the
> points taken could not be solved by a warning message/documentation with the
> option to refine later and on separate bugs?

We support this approach. It this stage we have this live patched because we couldn't keep waiting for the fix to come in a minor point update, and it's been a real saviour for us.
Comment 64 Katrin Fischer 2025-07-28 18:56:43 UTC
We have my and Cornelius sign-off, we are both from BSZ, so it cannot count for both signoff and QA. If we got more sign-offs I would be willing to put this in PQA (former RM get to QA)
Comment 65 Aleisha Amohia 2025-07-28 20:22:51 UTC
I had offered a signoff for the original patch. I don't know about the FOR DISCUSSION patch - should that be tested?
Comment 66 Magnus Enger 2025-07-29 07:31:41 UTC
Created attachment 184784 [details] [review]
Bug 39906: Add a simple quick Apache-level bot challenge

This patch adds a fast simple bot challenge via Apache.

To test:
0. Apply the patch
1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/.
2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/.
3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl
4. Note an instant response with a very simple HTML response
5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl
6. Note a very brief loading animation before the OPAC search page appearing
7. Do a search
8. Note no more loading animation appears
9. Close the incognito browser
10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl
11. Do a search via the "Library catalog" search bar
12. Note no loading animation appears

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Cornelius Hertfelder <cornelius.hertfelder@bsz-bw.de>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
We have tested this in production on at least three live sites for
several weeks, and it has worked as expected. We have not noticed any
negative side effects.
Comment 67 Marcel de Rooy 2025-07-30 06:31:18 UTC
Is there still any feedback about the alternative patch of Paul?
Comment 68 David Cook 2025-07-30 07:05:53 UTC
(In reply to Marcel de Rooy from comment #67)
> Is there still any feedback about the alternative patch of Paul?

Previously, I hadn't tried Paul's Javascript, but I did think that it looked very interesting.

It looks like the patch was attached in the wrong order but that's an easy fix at "git bz apply" time...

From a UX perspective, it looks similar, although it seems to take 4-5 seconds instead of 1 second. As a human user, I found that off-putting, but I think we could probably tune that. 

Overall, I think waiting more than 1-2 seconds is probably going to be unpleasant from a human UX perspective. 

--

I suppose my overall feedback would be... probably needs a bit more testing.

Also, personally I'm opposed to including opac-main.pl as I think it could have a negative impact on legitimate web indexing, but that's just my opinion. Maybe it's better to be conservative and include it as a baseline.
Comment 69 David Cook 2025-07-30 07:08:28 UTC
(In reply to Marcel de Rooy from comment #58)
> The only problem sofar is that they are too lazy to run your code ;)

Correct 😅
Comment 70 David Cook 2025-07-30 07:08:58 UTC
(In reply to David Cook from comment #69)
> (In reply to Marcel de Rooy from comment #58)
> > The only problem sofar is that they are too lazy to run your code ;)
> 
> Correct 😅

Not sure if you meant the humans or the bots, but as a human I'll own that one hehe.
Comment 71 Marcel de Rooy 2025-07-30 12:59:20 UTC
Created attachment 184886 [details] [review]
Bug 39906: Add a simple quick Apache-level bot challenge

This patch adds a fast simple bot challenge via Apache.

To test:
0. Apply the patch
1. sudo cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/.
2. sudo cp debian/templates/apache-shared-opac-plack.conf /etc/koha/.
3. curl http://localhost:8080/cgi-bin/koha/opac-search.pl
4. Note an instant response with a very simple HTML response
5. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-search.pl
6. Note a very brief loading animation before the OPAC search page appearing
7. Do a search
8. Note no more loading animation appears
9. Close the incognito browser
10. Open an incognito browser and go to http://localhost:8080/cgi-bin/koha/opac-main.pl
11. Do a search via the "Library catalog" search bar
12. Note no loading animation appears

Signed-off-by: Katrin Fischer <katrin.fischer@bsz-bw.de>
Signed-off-by: Cornelius Hertfelder <cornelius.hertfelder@bsz-bw.de>
Signed-off-by: Magnus Enger <magnus@libriotech.no>
We have tested this in production on at least three live sites for
several weeks, and it has worked as expected. We have not noticed any
negative side effects.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 72 Marcel de Rooy 2025-07-30 12:59:23 UTC Comment hidden (obsolete)
Comment 73 Marcel de Rooy 2025-07-30 13:02:16 UTC
Will move the patch from Paul to 40553 for further testing and consideration.
Refer the RM to comment33. Note that actually only comment4 has been done (and not by the initial author ;)
The feature has been tested well enough now.
Comment 74 Marcel de Rooy 2025-07-30 13:27:58 UTC
Created attachment 184890 [details] [review]
Bug 39906: (QA follow-up) Feature needs to be enabled specifically

See also comment33, point 4.

Test plan:
cp debian/templates/apache-shared-opac-antibot.conf /etc/koha/
Test enabling/disabling by toggling the comment of the SetEnvIf line,
restart apache and remove the KOHA_INIT and CGISESSID cookie.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 75 Paul Derscheid 2025-07-30 14:26:13 UTC
Thanks for moving my patch, Marcel.

The delay introduced by the additional code is indeed substantial. I experimented with this on a staging instance that was being bombarded with traffic at the time.

But doesn't help much if the clients don't even run the JS, that is very true :D
Comment 76 David Cook 2025-07-31 00:12:32 UTC
(In reply to Paul Derscheid from comment #75)
> The delay introduced by the additional code is indeed substantial. I
> experimented with this on a staging instance that was being bombarded with
> traffic at the time.

Ahh right. That makes sense. Let's talk about it more on bug 40553?
Comment 77 Marcel de Rooy 2025-08-22 08:08:39 UTC
+    requestAnimationFrame(() => {
+        const delay = performance.now() - timestamp1;
+        if (delay < 20){
+            timeout_incr += 1000;
+        }

David, why 20?
It seems that a common refresh rate of 60Hz will lead to values being under 16,7 with an average under 8,35. Could incidentally be 0.
Comment 78 David Cook 2025-08-25 00:15:20 UTC
(In reply to Marcel de Rooy from comment #77)
> +    requestAnimationFrame(() => {
> +        const delay = performance.now() - timestamp1;
> +        if (delay < 20){
> +            timeout_incr += 1000;
> +        }
> 
> David, why 20?
> It seems that a common refresh rate of 60Hz will lead to values being under
> 16,7 with an average under 8,35. Could incidentally be 0.

Hmm, I can't remember now. It doesn't really do any harm, but it doesn't do much good either.
Comment 79 Wainui Witika-Park 2025-08-26 01:39:13 UTC
Applied to 22.11.x-security for 22.11.30
Comment 80 Paul Derscheid 2025-08-27 08:52:03 UTC
Applied to 25.05.x-security for 25.05.03.