Bug 3144 - anti-spam for opac-suggestions
Summary: anti-spam for opac-suggestions
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: MJ Ray (software.coop)
QA Contact:
URL:
Keywords:
Depends on:
Blocks: 15035
  Show dependency treegraph
 
Reported: 2009-04-20 01:44 UTC by Chris Cormack
Modified: 2017-06-14 22:11 UTC (History)
8 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bugfix (6.72 KB, patch)
2009-06-22 11:36 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 3144 - anti-spam for opac-suggestions (3.09 KB, patch)
2014-05-28 00:33 UTC, Mason James
Details | Diff | Splinter Review
Bug 3144 - anti-spam for opac-suggestions (3.14 KB, patch)
2014-05-28 01:25 UTC, Mason James
Details | Diff | Splinter Review
suggestions form, with hidden 'negcap' field showing (82.98 KB, image/png)
2014-05-28 01:31 UTC, Mason James
Details
Bug 3144 - anti-spam for opac-suggestions (2.93 KB, patch)
2014-05-28 02:02 UTC, Mason James
Details | Diff | Splinter Review
suggestions form, with hidden 'negcap' field showing (73.69 KB, image/png)
2014-05-28 02:05 UTC, Mason James
Details
Bug 3144 - anti-spam for opac-suggestions (2.95 KB, patch)
2014-06-04 22:57 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 3144 - anti-spam for opac-suggestions (3.09 KB, patch)
2014-06-06 14:21 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Chris Cormack 2010-05-21 01:06:59 UTC


---- Reported by nicolas.morin@biblibre.com 2009-04-20 13:44:31 ----

if the opac-suggestions.pl?op=add page is accessible to anonymous users (AnonSuggestions syspref ON), we should provide a catpcha. Currently we don't, and spam coming through the form quickly forces the library to close down the option.
I suggest http://search.cpan.org/dist/Captcha-reCAPTCHA/lib/Captcha/reCAPTCHA.pm
It works well and has something to do with libraryland.



---- Additional Comments From mjr@ttllp.co.uk 2009-04-20 13:57:14 ----

Please add anti-spam tests like blogspam.net or even simple heuristics to defeat spam.  A sample of the spam you're getting would be useful.

Please do not add reCAPTCHA.  It discriminates against humans with visual or hearing impairments.

Retitling bug.




---- Additional Comments From jeanandre.santoni@biblibre.com 2009-06-16 14:12:15 ----

reCAPTCHA only discriminates against humans with both visual and
hearing impairments.




---- Additional Comments From mjr@ttllp.co.uk 2009-06-16 14:20:10 ----

Hearing impairments sufficient to prevent reCAPTCHA use are pretty common (estimates over 40% are common).

Please implement anti-spam in preference to reCAPTCHA.  CAPTCHAs are not anti-spam and I suspect most implementations violate disability discrimination laws.




---- Additional Comments From jeanandre.santoni@biblibre.com 2009-06-17 15:11:37 ----

Blogspam.net doesn't seem to work so well.
Take a look at my log (I Data::Dumped my request and their result):

$VAR1 = 'comment';
$VAR2 = 'La soupe à la fourchette Jean-Angleade 1987 Poche ';
$VAR3 = 'ip';
$VAR4 = '88.191.19.91';
SPAM:SpamBayes at /home/kivutar/Workspace/koha30x/C4/Suggestions.pm line 349.

According to blogspam.net, "La soupe à la fourchette Jean-Angleade 1987 Poche" is a spam...

Do you know another service? Or maybe I just used it wrong.



---- Additional Comments From mjr@ttllp.co.uk 2009-06-18 14:31:58 ----

I asked about this and was told it's essentially a result of using French-language data with one of the plugins that's trained with English-language ham data:

"I train the bayasian filter using results that are known-good or
 known-spam, but in both cases the training is with English text.

  I'd never considered the use of foreign language to be an issue,
 but I can easily imagine if the corpus is always trained with English
 that French wouldn't be seen, and is more likely to weigh in as spam.

  If this is a common occurrence then the caller should disable the
 bayasian plugin when they submit their comments for testing, see
 here for details of how to do that:

    http://blogspam.net/api/testComment.html

  (You'll want to disable the plugin called "bayasian".)"

Alternatively, Francophones may like to run their own spam filter, trained with French-language ham.  

(By the way, did you know that the documents used for reCAPTCHA are all taken from US sources at the moment?  I hear it's great fun for UKUS and NZUS spelling differences.)




---- Additional Comments From jeanandre.santoni@biblibre.com 2009-06-22 11:36:32 ----

Created an attachment
Bugfix

Bugfix



---- Additional Comments From mjr@ttllp.co.uk 2009-06-22 11:43:35 ----

Mollom seems to discriminate against visually-impaired users with non-CAPTCHA
eyetests if not thought to be ham and the free service only allows 100 comments
per day - see http://mollom.com/pricing

Therefore, integrating this Mollom patch would cause both a new accessibility
error and open an obvious denial of service attack opportunity.  Please make
this a contrib only.

Also, it's spelt "anonymous" not "anonimous".




---- Additional Comments From jeanandre.santoni@biblibre.com 2009-06-22 12:17:43 ----

The use I made of Mollom does not discriminate against visually-impaired users. It is just a filter like blogspam.net. Except that it works better.

Can you tell me more about the denial of service attack opportunity please?



---- Additional Comments From gmcharlt@gmail.com 2009-06-29 15:12:46 ----

Comment on proposed patch:

My main concern with this patch is that there needs to be an
abstraction layer, no matter how thin, for spam filtering before we
start adding in support for specific services like Mollom.  A simple
class in C4 that can delegate a spam/ham decision should be
sufficient.  We have a big enough mess with the current state of
support for enhanced content providers without any kind of abstraction
layer, so I'd like to take this opportunity for us to not fall into a
similar trap.



--- Bug imported by chris@bigballofwax.co.nz 2010-05-21 01:06 UTC  ---

This bug was previously known as _bug_ 3144 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=3144
Imported an attachment (id=1127)

Actual time not defined. Setting to 0.0
The original reporter of this bug does not have
   an account here. Reassigning to the person who moved
   it here: chris@bigballofwax.co.nz.
   Previous reporter was nicolas.morin@biblibre.com.
The original assignee of this bug does not have
   an account here. Reassigning to the default assignee
   for the component, chris@bigballofwax.co.nz.
   Previous assignee was jeanandre.santoni@biblibre.com.
Bug reassigned, setting status to "NEW".
   Previous status was "ASSIGNED".
CC member mjr@ttllp.co.uk does not have an account here
The original submitter of attachment 1127 [details] [review] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.

Comment 1 Owen Leonard 2010-06-29 18:52:44 UTC
I'm removing "PATCH-sent" status from this bug since the correct resolution is still under discussion.
Comment 2 MJ Ray (software.coop) 2010-07-07 19:27:08 UTC
Restoring where "Previous assignee was mjr@ttllp.co.uk"
Comment 3 Galen Charlton 2013-08-12 15:54:04 UTC
Noting that a Koha user recently reported an instance of purchase suggestion spam:

http://lists.katipo.co.nz/pipermail/koha/2013-July/036898.html
Comment 4 Christopher Vella 2014-03-06 03:39:39 UTC
Hi everyone

I have been investigating this issue today for one of Calyx's clients. I decided to do a bit of reading up on the issue of spam protection.

reCAPTCHA was the first solution that came to mind, however valid points about accessibility issues have already been made.

What about the possibility of implementing a negative captcha? That way OPACs remain accessible. It would also mean we don't have to deal with the false positive detection of suggestions (as with analytical spam detection). The only downside I can see with this would be a bruteforce tool that tries ignoring particular fields or a custom coded spam bot (which seems very unlikely considering the value of the target).

I also like the http://visualcaptcha.net/ project.



Chris
Comment 5 Chris Cormack 2014-03-11 19:26:38 UTC
A negative captcha seems to be the current best practice, I vote we go with that.
Comment 6 Galen Charlton 2014-03-11 23:37:21 UTC
(In reply to Chris Cormack from comment #5)
> A negative captcha seems to be the current best practice, I vote we go with
> that.

Agreed.
Comment 7 Bob Birchall 2014-05-21 03:18:50 UTC
(In reply to Galen Charlton from comment #6)
> (In reply to Chris Cormack from comment #5)
> > A negative captcha seems to be the current best practice, I vote we go with
> > that.
> 
> Agreed.

A client has agreed to fund this.  We'll proceed with it.
Comment 8 Mason James 2014-05-28 00:33:52 UTC Comment hidden (obsolete)
Comment 9 Mason James 2014-05-28 01:25:57 UTC Comment hidden (obsolete)
Comment 10 Mason James 2014-05-28 01:31:12 UTC Comment hidden (obsolete)
Comment 11 Mason James 2014-05-28 02:02:43 UTC Comment hidden (obsolete)
Comment 12 Mason James 2014-05-28 02:05:32 UTC
Created attachment 28508 [details]
suggestions form, with hidden 'negcap' field showing
Comment 13 Mason James 2014-05-28 02:06:20 UTC
patch updated for bootstrap, not prog theme :)
Comment 14 Christopher Brannon 2014-06-04 22:55:51 UTC
Works great.  I recommend updating the other themes with this function as well.

Christopher
Comment 15 Biblibre Sandboxes 2014-06-04 22:56:49 UTC
Patch tested with a sandbox, by Christopher Brannon <cbrannon@cdalibrary.org>
Comment 16 Biblibre Sandboxes 2014-06-04 22:57:10 UTC Comment hidden (obsolete)
Comment 17 Chris Cormack 2014-06-04 22:59:04 UTC
(In reply to Christopher Brannon from comment #14)
> Works great.  I recommend updating the other themes with this function as
> well.
> 
> Christopher

The other themes were deprecated in 3.14.0 and will be removed in 3.18.0

So you would probably have to fund someone to backport this to them if you wanted.
Comment 18 Christopher Brannon 2014-06-04 23:02:23 UTC
Do you think it should be communicated to the user that the submission failed?  I know it is highly unlikely, but in the off chance that it is a human, and something goes wrong, should something be in place?

Christopher
Comment 19 Chris Cormack 2014-06-04 23:04:31 UTC
(In reply to Christopher Brannon from comment #18)
> Do you think it should be communicated to the user that the submission
> failed?  I know it is highly unlikely, but in the off chance that it is a
> human, and something goes wrong, should something be in place?
> 
Perhaps, however id be worried that telling them it failed might make the bot try again and again.
Comment 20 Jonathan Druart 2014-06-06 14:21:55 UTC
Created attachment 28701 [details] [review]
Bug 3144 - anti-spam for opac-suggestions

this patch adds a negative-captcha feature to the purchase suggestions form

some info for the curious...
 http://nedbatchelder.com/text/stopbots.html
 https://github.com/subwindow/negative-captcha
 http://www.rubydoc.info/github/subwindow/negative-captcha/frames

to test this patch...

1/ apply patch

2/ enable 'suggestion' and 'AnonSuggestions' sysprefs

3/ attempt to add a 'purchase suggestion' in OPAC
   adding the suggestion should succeed

4/ edit the koha-tmpl/opac-tmpl/bootstrap/en/modules/opac-suggestions.tt file,
    to reveal the hidden 'negcap' field

   replace line 87 opac-suggestions.tt, from...
     <li id="negcap" style="position: absolute; left: -2000px;">
   to ...
     <li id="negcap">

5/ attempt to add another suggestion, and populate the 'negcap' field
   adding the suggestion should fail

Signed-off-by: Christopher Brannon <cbrannon@cdalibrary.org>
Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Amended patch: Remove the bug report number in the comment since git log
cans provide the same result.
Comment 21 Chris Cormack 2014-06-08 09:13:14 UTC
Simple patch, easy to test, pushed to Master
Comment 22 Bob Birchall 2014-06-09 07:00:25 UTC
(In reply to Chris Cormack from comment #21)
> Simple patch, easy to test, pushed to Master

This enhancement has been sponsored by Professional Support Coordinators' Alliance, which provides professional support and training to the child care and early learning sector in Australia.
Comment 23 Galen Charlton 2014-06-09 16:14:10 UTC
Given that some libraries have been significantly affected by purchase suggestion spam and consequently have not been able to use AnonSuggestions, I'm considering this a bugfix.

Therefore, pushed to 3.16.x for inclusion in 3.16.1.
Comment 24 David Cook 2015-02-18 02:59:32 UTC
Chatted with Chris Cormack, Chris Vella, and Mason about this a bit on IRC.

Screen readers will still read the 'negcap' field, which is suboptimal.

However, even if a screenreader does read the field, hopefully people will still leave this blank. 

Perhaps it might be an idea to include a more obvious label name like "do not fill me in".

Anyway, just leaving some links here. Chris Vella found the one about "aria-hidden". That could be really good to add.


Relevant links:

http://www.w3.org/TR/wai-aria/states_and_properties#aria-hidden

http://www.visionaustralia.org/living-with-low-vision/learning-to-live-independently/using-technology-and-computers/blog---accessibility-and-assistive-technology-blog/blog/accessibility-blog/2014/12/09/effective-alternatives-to-inaccessible-captchas

http://webaim.org/techniques/css/invisiblecontent/#absolutepositioning
Comment 25 Katrin Fischer 2015-03-25 22:29:24 UTC
A human being might still misuse the feature - I think it should be a choice  for the library, so they can turn off 'non logged in cart sending'.
Comment 26 Katrin Fischer 2015-03-25 22:35:24 UTC
Last comment was meant for another bug... bug 4274
Comment 27 Mason James 2015-10-19 23:45:37 UTC
Hmm, I think I've found a small bug here, patch coming...
Comment 28 David Cook 2016-02-03 05:04:31 UTC
I think that this was working for a time, but it appears that some bots may have wised up...

I'm noticing spam coming through once again, which I assume means that someone is having their bot ignore the "negcap" field or fields that are located off screen. Both would be easy to do.

We're going to look at blocking some IP addresses to start, but this one is a distributed spamming of the system so that will likely only be partially successful. 

I have a couple curl commands that I could use to easily spam opac-suggestions.pl...
Comment 29 David Cook 2016-02-03 05:13:21 UTC
You know... we could do a text-based positive captcha. That should be accessible.

Something like... "write the third word from the previous paragraph in the following box". 

A human should be able to understand that instruction, and it should require  fairly sophisticated natural language processing for a bot to figure out.
Comment 30 David Cook 2016-02-03 05:13:40 UTC
Ideally, it wouldn't always be the third word. It would sometimes be the first word or the last word of the paragraph. Something like that...
Comment 31 Christopher Vella 2016-02-03 06:01:29 UTC
Potentially the bot could be trying input forms at random, or more advanced and only reading through the visible elements of the page.

Rather than go down the positive captcha route yet. We could attempt to make this negcaptcha smarter. One idea could be to IP ban any user who submits content to that form (I don't see any reason why a normal user would). We could also have the field name randomise per instance.
Comment 32 Marc Véron 2016-02-03 10:55:24 UTC
The self registration ( cgi-bin/koha/opac-memberentry.pl ) has already a text based captcha with a random string:
"Please type the following characters into the preceding box: ODXZX"

I think it would be good to re-use technique/code at all places where we have 'human user input'.
Comment 33 Mason James 2016-02-04 05:18:44 UTC
(In reply to Mason James from comment #27)
> Hmm, I think I've found a small bug here, patch coming...

update: i added the patch to a new SEC bug

 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15035
Comment 34 Mason James 2016-02-04 05:20:06 UTC
(In reply to David Cook from comment #28)
> I think that this was working for a time, but it appears that some bots may
> have wised up...
> 

hiya, are you running this follow-up patch?

 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15035
Comment 35 David Cook 2016-02-08 01:15:20 UTC
(In reply to Mason James from comment #34)
> hiya, are you running this follow-up patch?
> 
>  https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15035

Alas, I don't have access to see that security bug :/.
Comment 36 David Cook 2016-02-08 01:18:14 UTC
(In reply to Christopher Vella from comment #31)
> Potentially the bot could be trying input forms at random, or more advanced
> and only reading through the visible elements of the page.
> 

Yeah, I think there are a few different things it could be. I'd love to know what data is being sent exactly.

> Rather than go down the positive captcha route yet. We could attempt to make
> this negcaptcha smarter. 

Totally agree. I'm absolutely on board for that.

> One idea could be to IP ban any user who submits
> content to that form (I don't see any reason why a normal user would). 

I'm not sure I understand what you mean here. When the user fills out the form, their browser would be POSTing the data to the Koha server, so it would be something a normal user would do. 

> We could also have the field name randomise per instance.

Maybe. I'd be interested to see how other people are solving this problem out in the world.
Comment 37 Katrin Fischer 2016-02-08 07:35:02 UTC
(In reply to David Cook from comment #35)
> (In reply to Mason James from comment #34)
> > hiya, are you running this follow-up patch?
> > 
> >  https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15035
> 
> Alas, I don't have access to see that security bug :/.

Fixed :)
Comment 38 Christopher Vella 2016-02-08 23:41:31 UTC
> > One idea could be to IP ban any user who submits
> > content to that form (I don't see any reason why a normal user would). 
> 
> I'm not sure I understand what you mean here. When the user fills out the
> form, their browser would be POSTing the data to the Koha server, so it
> would be something a normal user would do. 

Normal users shouldn't be submitting any data to the negcap field (because its not visible). Therefore only spammers should be submitting data too it. If we introduced a banning system, their subsequent spamming attempts would be stopped.
Comment 39 David Cook 2016-02-09 01:59:34 UTC
(In reply to Christopher Vella from comment #38)
> Normal users shouldn't be submitting any data to the negcap field (because
> its not visible). Therefore only spammers should be submitting data too it.
> If we introduced a banning system, their subsequent spamming attempts would
> be stopped.

Ah yes. I understand you now. Yeah, that would lower the amount of spam requests you're getting.

It seems that we're getting POSTs without the negcap field filled in, which is how we're still getting spam suggestions, but with your idea they may have been banned before they figured that out...
Comment 40 David Cook 2016-02-09 02:00:20 UTC
(In reply to Mason James from comment #34)
> hiya, are you running this follow-up patch?
> 
>  https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=15035

Not running the follow-up patch, but that doesn't appear to be an issue for 3.14 :/