Bug 18615 - LDAP configuration error causes server resource depletion under Plack
Summary: LDAP configuration error causes server resource depletion under Plack
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Authentication (show other bugs)
Version: 16.11
Hardware: HP Linux
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-05-16 19:23 UTC by Sven Coenye
Modified: 2017-07-26 16:18 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Sven Coenye 2017-05-16 19:23:52 UTC
When running with Plack enabled, it is possible to get KOHA to keep spawning processes until the hosts's resources are exhausted. The immediate upshot is that KOHA hangs, the final result is that all other services on the host eventually stop functioning as well.

Steps to reproduce:
Enable useldapserver and specify an empty search base in the ldapserver element:
<useldapserver>1</useldapserver>
<ldapserver>
  ...
  <base></base>
  ...
</ldapserver>

Restart koha-plack for the instance.

The plack-error.log file starts filling up with "Compilation failed" errors on all modules that somehow invoke Auth_with_ldap.pm. The "top" command shows two active /etc/koha/plack processes owned by the koha-$instance user. Each process uses ~50% of the available CPU cycles. These processes are replaced continuously. ps aux | grep apache shows an every increasing number of Apache instances owned by the same user.

Without Plack, the same configuration problem triggers an error page for KOHA, but the rest of the services on the host are not affected.
Comment 1 Mason James 2017-05-17 02:16:00 UTC
(In reply to Sven Coenye from comment #0)
> When running with Plack enabled, it is possible to get KOHA to keep spawning
> processes until the hosts's resources are exhausted. The immediate upshot is
> that KOHA hangs, the final result is that all other services on the host
> eventually stop functioning as well.
> 
> Steps to reproduce:
> Enable useldapserver and specify an empty search base in the ldapserver
> element:
> <useldapserver>1</useldapserver>
> <ldapserver>
>   ...
>   <base></base>
>   ...
> </ldapserver>
> 

curious.. does the error only occur if the <base> element is empty?
Comment 2 Sven Coenye 2017-05-17 14:21:04 UTC
The condition can be reached by triggering any of the "or die" clauses at the upper level of the Auth_with_ldap module. E.g. a blank hostname will also trigger it.

More info:
Although we originally ran into this on our production server, we have been able to reproduce this on a fresh KOHA install. Both hosts run Debian 8. The production server is a SysV virtual machine, the test server is bare metal using systemd. KOHA is the only application installed on the test machine.
Comment 3 Sven Coenye 2017-07-17 18:46:19 UTC
2017.05 has exposed a addition way to trigger this condition.

2016.11 and prior did not need defined values for <user> and <pass> in the ldapserver configuration block in koha-conf.xml. A change in Auth_with_ldap.pm (see bug 18947) causes KOHA to attempt an anonymous bind unless user and pass are defined, regardless of whether these values will be used or not. This causes Active Directory to reject the attempted login. This rejection is unexpected and is trapped by a "die" clause, causing Plack to enter the death spiral.
Comment 4 Sven Coenye 2017-07-26 16:18:11 UTC
On second look, the consequences here are not quite as severe. As the "die" clause is only triggered when the login fails, Plack does not enter the infinite loop. 

It does, however, leave a handful of defunct Apache instances behind for each such failed login and those will eventually cause the same resource depletion as the loop case.