On Ubuntu, output of /usr/share/koha/bin/cronjobs/process_message_queue.pl in an email from cron: ourlib: Can't locate Authen/CAS/Client/Response/Failure.pm: ./Authen/CAS/Client/Response/Failure.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. ourlib: ...propagated at /usr/share/perl/5.22/base.pm line 106. ourlib: BEGIN failed--compilation aborted at /usr/share/perl5/Authen/CAS/Client/Response.pm line 68. ourlib: Compilation failed in require at /usr/share/perl5/Authen/CAS/Client.pm line 8. . . . process_message_queue.pl is currently invoked from /etc/cron.d/koha-common I don't find any Failure.pm on the system. This only started happening after a recent update which resulted in Bug 17234. Not sure if issues with that may have also affected this. Have done a full reinstall (without Purge) of the packages.
Created attachment 57947 [details] Full error as displayed in the email
After extracting Failure to its own file, process_message_queue.pl runs without a peep. I don't understand enough of Perl to know what's going on. Files end up like this: root@koha:/usr/share/perl5/Authen/CAS/Client# find . ./Response.pm.bak ./Response ./Response/Failure.pm ./Response.pm Failure.pm is cut out of Response.pm: (didn't have much success with multi-line comments!) root@koha:/usr/share/perl5/Authen/CAS/Client# cat Response/Failure.pm require 5.006_001; use strict; use warnings; #====================================================================== # Authen::CAS::Client::Response::Failure # package Authen::CAS::Client::Response::Failure; use base qw/ Authen::CAS::Client::Response /; sub _ATTRIBUTES () { code => undef, message => '', $_[0]->SUPER::_ATTRIBUTES } sub new { my $class = shift; $class->SUPER::new( @_, _ok => 0 ) } sub code { my ( $self ) = @_; $self->{code} } sub message { my ( $self ) = @_; $self->{message} }
After separating Failure.pm, the emails are now showing the same problem with Success.pm. After extracting Success.pm in the same way, now it complains about Exceptions.pm : Can't locate Koha/Exceptions/Exception.pm: ./Koha/Exceptions/Exception.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97, <DATA> line 755 Seems like a problem with the build, or (the current version of?) perl not supporting multiple classes within a .pm file?
Futher, after separating Exception.pm, we get: ourlib: Class Koha::Exceptions::Exception appears to be a typo as it is only specified in the 'isa' param for Koha::Exceptions::Virtualshelves::InvalidInviteKey ourlib: BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Exceptions.pm line 41, <DATA> line 755. . . Exceptions.pm itself has many references to Exception (I didn't separate all of them). Anyhow, I think all the above problems are just a source of one very small thing which I'm not seeing / understanding. I doubt the classes needed to be separated out...
Separating Exception.pm seemed to have broken other things, so I've backed out of it.
I get pretty much the exact same error when trying a koha-rebuild-zebra, so I think this is a bit more generic than the message queue. The error occurs in a file not supplied by Koha, and when called from a core perl file for perl 5.22. It seems 5.18 (the perl version from Ubuntu 14.04) does things differently than 5.22 does. It all looks weird, because the CAS code from Ubuntu 14.04 is pretty much the same as the one from 16.04, so that method of using multiple modules in a single file seems to have worked better back then. I think I'll have to back out to 14.04 again for the time being, without being able to run plack.
Can confirm this is happening with Ubuntu 16.04 (as the webserver db on another server) and Koha 16.11.05 No mail is sent with this bug Can't locate Authen/CAS/Client/Response/Failure.pm: ./Authen/CAS/Client/Response/Failure.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97.
On affected servers, changing the /etc/cron.d/koha-common job for process message queue to the following: */15 * * * * root cd /tmp && koha-foreach --enabled --email /usr/share/koha/bin/cronjobs/process_message_queue.pl seems to end run the issue. Thanks to Chris C. for the work around. As Chris says: <rangi> its a permissions issue, something in perl 5.22 has changed <rangi> its not actually a Koha problem, there are a few perl modules that have the same problem
(In reply to Liz Rea from comment #8) > On affected servers, changing the /etc/cron.d/koha-common job for process > message queue to the following: > > */15 * * * * root cd /tmp && koha-foreach --enabled --email > /usr/share/koha/bin/cronjobs/process_message_queue.pl > > seems to end run the issue. > > Thanks to Chris C. for the work around. > > As Chris says: > <rangi> its a permissions issue, something in perl 5.22 has changed > <rangi> its not actually a Koha problem, there are a few perl modules that > have the same problem Ah finally - a workaround!
What should we do about this?
We could do a patch that adds the cd /tmp && Otherwise we could try to fix the upstream module, to work better with perl 5.22 and cron. But yeah, it's an upstream problem, I think the module is pretty much unmaintained, maybe finding a better module is a better plan. Maybe the cd /tmp && for cron patch, and then work on a long term solution?
Note: this seems to be the same compilation error that was tripping up plack in Bug 16138
http://www.perlmonks.org/?node_id=1127880
Just a FYI on this. I found that *not* using koha-foreach, and instead use the actual koha instance user also works. For example, instead of: root koha-foreach --enabled --email /usr/share/koha/bin/cronjobs/process_message_queue.pl if you use: INST-koha /usr/share/koha/bin/cronjobs/process_message_queue.pl it works. So taking a look at how koha-foreach switches to the user might provide some clues. Can confirm this also happens on Debian 8.4.
So, what do we do? Do we release 17.05 as it or we propose a workaround? Is the `cd /tmp` trick added to the cron line acceptable?
(In reply to Jonathan Druart from comment #15) > So, what do we do? > Do we release 17.05 as it or we propose a workaround? > Is the `cd /tmp` trick added to the cron line acceptable? I think we should make workaround now and create new bug report for long term solution immedietely...
I am working on replacing this module with one that supports CAS 2.0 and doesnt have this problem. It won't obviously be ready in time though. So I will write a patch to add the cd /tmp && and attach it to here. Should be ready soon
(In reply to Larry Baerveldt from comment #14) > Just a FYI on this. I found that *not* using koha-foreach, and instead use > the actual koha instance user also works. > > For example, instead of: > root koha-foreach --enabled --email > /usr/share/koha/bin/cronjobs/process_message_queue.pl > > if you use: > INST-koha /usr/share/koha/bin/cronjobs/process_message_queue.pl > > it works. > > So taking a look at how koha-foreach switches to the user might provide some > clues. > > Can confirm this also happens on Debian 8.4. Yep thats because you are running the script as root then. Root has permission to write in that dir. With for-each you are running as the user, it doesn't. However the real problem is this module. Which we need to replace, the workaround should hold for then
Created attachment 63676 [details] [review] Bug 17717 : Work around for the permissions problem when running process_message_queue To test 1/ Make sure some things are in the message queue 2/ Run the cron jobs, notice nothing is sent 3/ Build and install a new package (or manually update the /etc/cron.d/koha-common file) 4/ Run the jobs again, notice mail is sent
Created attachment 63677 [details] [review] [SIGNED-OFF] Bug 17717 : Work around for the permissions problem when running process_message_queue To test 1/ Make sure some things are in the message queue 2/ Run the cron jobs, notice nothing is sent 3/ Build and install a new package (or manually update the /etc/cron.d/koha-common file) 4/ Run the jobs again, notice mail is sent Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
This bug really confuses me. (In reply to Chris Cormack from comment #18) > Yep thats because you are running the script as root then. Root has > permission to write in that dir. With for-each you are running as the user, > it doesn't. Why does Perl need write access to the current directory for loading a module ? > Can't locate Authen/CAS/Client/Response/Failure.pm: ./Authen/CAS/Client/Response/Failure.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. suggests that ./Authen/CAS/Client/Response/Failure.pm exists but is not readable. This file shouldn't exist as Authen::CAS::Client::Response::Failure is defined in Authen/CAS/Client/Response.pm. Why does it exist ? (In reply to Josef Moravec from comment #20) > To test > 1/ Make sure some things are in the message queue Why ? What could possibly happen during process_message_queue.pl that trigger an error related to a CAS module ? And why the solution to this problem is 'cd /tmp' ? :)
(In reply to Julian Maurice from comment #21) > This bug really confuses me. Same here :) Chris do you have the answers?
Ok The original report of the error describes the problem, so I don't think i need to go proving it exists? I'll do a summary anyway. With perl 5.22 with latest ubuntu, debian testing and above, etc have. Something has changed in the way it deals with modules that have multiple packages in them. This is not unique to this module, if you search for Permission denied at /usr/share/perl/5.22/base.pm line 97 You will see others doing it too EG I've noticed that Object::InsideOut v3.98 fails when the current directory is not readable by the user running the Perl binary. Most Perl modules don't have this requirement, and so OIO creates a very cryptic error message when you run it in such circumstances. However Authen::CAS::Client is a module we use, that is loaded by one of the modules that process_message_queue.pl uses. By adding cd /tmp && to the cron we make sure we avoid the not readable part. Of course running as root gets round this too. This is not a fix, like I said, but a simple workaround so that notices get sent in Koha on perl 5.22. Until this module is replaced (which it is going to be) or another solution involving changing how koha-for-each works under cron (it works fine on the command line) is done. This does only occur on perl 5.22 and up, so something changed in that.
I've made a fresh install on Debian stretch (Perl 5.24) but I can't reproduce the problem. root@koha-stretch:~# sudo -u koha sh -c 'touch test' touch: cannot touch 'test': Permission denied root@koha-stretch:~# sudo -u koha sh -c 'ls' ls: cannot open directory '.': Permission denied As you can see I have no read nor write access to this directory, but when I execute root@koha-stretch:~# sudo -u koha sh -c 'PERL5LIB=/home/koha/koha KOHA_CONF=/home/koha/etc/koha-conf.xml perl /home/koha/koha/misc/cronjobs/process_message_queue.pl' I have no errors and the message queue is processed. Can you tell me what I'm doing wrong ?
(In reply to Julian Maurice from comment #24) > > I have no errors and the message queue is processed. > > Can you tell me what I'm doing wrong ? You are not running it as cron, you are running it from the commandline. The error only manifests under cron.
# cat /etc/cron.d/koha * * * * * root sudo -u koha sh -c 'PERL5LIB=/home/koha/koha KOHA_CONF=/home/koha/etc/koha-conf.xml perl /home/koha/koha/misc/cronjobs/process_message_queue.pl > /tmp/koha.log 2>&1' * * * * * root sudo -u koha sh -c 'pwd > /tmp/pwd.log 2>&1' /tmp/pwd.log contains "/root" /tmp/koha.log is empty, and the message queue is processed
I feel like you are not reading the history of the bug, it is when using the cron job from the packages. You are not running that. Step 3 of the test plan says 3/ Build and install a new package (or manually update the /etc/cron.d/koha-common file) Not manually run some other cron job. I am going to bed now, and looks like it has missed the release, so you will have time to read up yourself.
I've read the history of the bug, many times, and I understand that the problem is with the package install, which I'm trying to mimic to understand what's happening. If my command succeeds, why the one with koha-foreach should fail ? Anyway I'll try with a package install later when I have more time
Just tried a package install following instructions on https://wiki.koha-community.org/wiki/Debian. Same result. The cron ran process_message_queue.pl and the message queue was processed. No errors. Should I have specific data in message_queue to trigger the error ?
Did you run it manually in your package installation or using the standard cron.d entry? */15 * * * * root koha-foreach --enabled --email /usr/share/koha/bin/cronjobs/process_message_queue.pl Looking at the history of the bug a lot of people have run into it by now - I think there must be a difference in how you test.
I let the cron do it, as Chris suggested in comment 25. The only thing I modified in the cron entry is '*/15' to get it launched every minute
Is there a difference between the standard koha-foreach with a sudo koha user and the sudo you added in your cron job ? Are the permissions of your koha user identical ?
(In reply to Marcel de Rooy from comment #32) > Is there a difference between the standard koha-foreach with a sudo koha > user and the sudo you added in your cron job ? > Are the permissions of your koha user identical ? I'm not sure what you are talking about. I used the /etc/cron.d/koha-common shipped with the debian package, and I didn't change the permissions of the 'koha1-koha' user created by koha-create. I didn't modify the sudoers file either (http://paste.koha-community.org/435)
More info: $ pmvers Authen::CAS::Client 0.07 $ pmvers Authen::CAS::Client::Response 0.03 $ perl -v This is perl 5, version 24, subversion 1 (v5.24.1) built for x86_64-linux-gnu-thread-multi $ lsb_release -ds Debian GNU/Linux 9.0 (stretch)
And what about the difference between perl 5.22 and 5.24 ?
In comment 23, Chris says: > With perl 5.22 with latest ubuntu, debian testing and above, etc have. > Something has changed in the way it deals with modules that have multiple > packages in them. and > This does only occur on perl 5.22 and up, so something changed in that. Since Debian testing have Perl 5.24 I assumed I could reproduce the problem with this version. I will test with ubuntu 16.10 which has Perl 5.22. Let's hope I reproduce the problem this time :)
Same with Ubuntu 16.10/Perl 5.22/koha-common 16.11.08 I'm running out of ideas... Does any other QAer want to try ? :)
I can reproduce a similar error with this command: root@koha-yakkety:~# sudo -u koha1-koha PERL5LIB=. perl -MAuthen::CAS::Client -E 'say "Hi"' Can't locate Authen/CAS/Client.pm: ./Authen/CAS/Client.pm: Permission denied. BEGIN failed--compilation aborted. root@koha-yakkety:~# cd /tmp/ root@koha-yakkety:/tmp# sudo -u koha1-koha PERL5LIB=. perl -MAuthen::CAS::Client -E 'say "Hi"' Hi So, people that can reproduce the problem, could you please paste the output of the following command (at least the @INC part): # As root sudo -u <KOHA_USER> perl -V
Ok, I managed to reproduce the conditions that trigger the bug. GOOD NEWS: It is fixed in master, 16.11 and 16.05 already, as a side effect from bug 16733. The problem was in koha-foreach code, that is not using koha-shell but crafting its own env instead. To reproduce the problem, as a root user run: root$ echo $PERL5LIB <empty> root$ . /usr/share/koha/bin/koha-functions.sh root$ adjust_paths_dev_install kohadev root$ echo $PERL5LIB /home/vagrant/kohaclone So, if adjust_paths_dev_install is not ran, and the command is run by root, PERL5LIB is empty, and perl uses the fallback for include (i.e. '.'). This are the conditions in which cron.d/koha-common runs process_message_queue.pl. This happened in koha-foreach code, before bug 16733. This is the relevant code in debian/scripts/koha-foreach: for name in $(koha-list $listopts) do # Optionally use alternative paths for a dev install adjust_paths_dev_install $name cmd=`echo "$@" | sed -e s/__instancename__/${name}/g` ( exec 3>&1 sudo -u "$name-koha" \ env PERL5LIB=$PERL5LIB \ KOHA_CONF="/etc/koha/sites/$name/koha-conf.xml" \ ${cmd} 2>&1 >&3 | sed -e "s/^/$name: /" >&2 3>&- exec 3>&- ) | sed -e "s/^/$name: /" done So before the 16733 backport, PERL5LIB was incorrectly set. This script should definitely run koha-shell -c "${cmd}" $name instead of crafting all the environment variables.
This issue is NOT fixed in 17.05. I'm seeing this now after having upgraded from 16.11. However, I've discovered some things that might be helpful. As Chris mentioned, this appears to be a bug with base.pm (IMO). Perhaps it's behavior has changed due to this security resolution that forces Perl to die if a required module is cannot be read when traversing @INC: https://rt.perl.org/Public/Bug/Display.html?id=113422 According to this post, base.pm may attempt to require a module that does not exist and traverse the entire @INC array: https://rt.cpan.org/Public/Bug/Display.html?id=106552#txn-1589169 Evidently, it dies as soon as it encounters a directory it cannot read. Avoiding '.' in PERL5LIB doesn't really help since Perl already includes '.' in @INC (also a security concern???). Working around this means either ensuring that '.' points to a directory that can be read (e.g. "cd /tmp") or modifying the @INC array to remove '.' from it. The latter seems better so I modified the BEGIN block in /usr/share/koha/bin/kohalib.pl like this: ------------------------------------- use strict; #use warnings; FIXME - Bug 2505 my $module_dir; BEGIN { $module_dir = '/usr/share/koha/lib'; die if $module_dir =~ /^[_]{2}PERL_MODULE_DIR[_]{2}$/; @INC = grep {$_ ne '.'} @INC; } use lib $module_dir; 1; ------------------------------------- This also allows the change to be in one place and avoid having to insert a 'cd /tmp' for each cron job. I'm running Perl 5.22 - I wonder if it's fixed in 5.24.
(In reply to Marco Moreno from comment #40) > This issue is NOT fixed in 17.05. I'm seeing this now after having upgraded > from 16.11. However, I've discovered some things that might be helpful. Can you mention your steps to reproduce?
ubuntu@koha:~$ sudo -i root@koha:~# sudo -u library-koha perl -MAuthen::CAS::Client -e 'print "Ok\n"' Can't locate Authen/CAS/Client/Response/Failure.pm: ./Authen/CAS/Client/Response/Failure.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106. BEGIN failed--compilation aborted at /usr/share/perl5/Authen/CAS/Client/Response.pm line 68. Compilation failed in require at /usr/share/perl5/Authen/CAS/Client.pm line 8. BEGIN failed--compilation aborted at /usr/share/perl5/Authen/CAS/Client.pm line 8. Compilation failed in require. BEGIN failed--compilation aborted. root@koha:~# cd /tmp root@koha:/tmp# sudo -u library-koha perl -MAuthen::CAS::Client -e 'print "Ok\n"' Ok root@koha:~# perl -Mbase -e 'print "$base::VERSION\n"' 2.22 root@koha:~# perl -V Summary of my perl5 (revision 5 version 22 subversion 1) configuration: Platform: osname=linux, osvers=3.16.0, archname=x86_64-linux-gnu-thread-multi uname='linux localhost 3.16.0 #1 smp debian 3.16.0 x86_64 gnulinux ' config_args='-Dusethreads -Duselargefiles -Dcc=x86_64-linux-gnu-gcc -Dcpp=x86_64-linux-gnu-cpp -Dld=x86_64-linux-gnu-gcc -Dccflags=-DDEBIAN -Wdate-time -D_FORTIFY_SOURCE=2 -g -O2 -fstack-protector-strong -Wformat -Werror=format-security -Dldflags= -Wl,-Bsymbolic-functions -Wl,-z,relro -Dlddlflags=-shared -Wl,-Bsymbolic-functions -Wl,-z,relro -Dcccdlflags=-fPIC -Darchname=x86_64-linux-gnu -Dprefix=/usr -Dprivlib=/usr/share/perl/5.22 -Darchlib=/usr/lib/x86_64-linux-gnu/perl/5.22 -Dvendorprefix=/usr -Dvendorlib=/usr/share/perl5 -Dvendorarch=/usr/lib/x86_64-linux-gnu/perl5/5.22 -Dsiteprefix=/usr/local -Dsitelib=/usr/local/share/perl/5.22.1 -Dsitearch=/usr/local/lib/x86_64-linux-gnu/perl/5.22.1 -Dman1dir=/usr/share/man/man1 -Dman3dir=/usr/share/man/man3 -Dsiteman1dir=/usr/local/man/man1 -Dsiteman3dir=/usr/local/man/man3 -Duse64bitint -Dman1ext=1 -Dman3ext=3perl -Dpager=/usr/bin/sensible-pager -Uafs -Ud_csh -Ud_ualarm -Uusesfio -Uusenm -Ui_libutil -Uversiononly -DDEBUGGING=-g -Doptimize=-O2 -dEs -Duseshrplib -Dlibperl=libperl.so.5.22.1' hint=recommended, useposix=true, d_sigaction=define useithreads=define, usemultiplicity=define use64bitint=define, use64bitall=define, uselongdouble=undef usemymalloc=n, bincompat5005=undef Compiler: cc='x86_64-linux-gnu-gcc', ccflags ='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64', optimize='-O2 -g', cppflags='-D_REENTRANT -D_GNU_SOURCE -DDEBIAN -fwrapv -fno-strict-aliasing -pipe -I/usr/local/include' ccversion='', gccversion='5.3.1 20160311', gccosandvers='' intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3 d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3 ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8 alignbytes=8, prototype=define Linker and Libraries: ld='x86_64-linux-gnu-gcc', ldflags =' -fstack-protector-strong -L/usr/local/lib' libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/5/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib libs=-lgdbm -lgdbm_compat -ldb -ldl -lm -lpthread -lc -lcrypt perllibs=-ldl -lm -lpthread -lc -lcrypt libc=libc-2.21.so, so=so, useshrplib=true, libperl=libperl.so.5.22 gnulibc_version='2.21' Dynamic Linking: dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E' cccdlflags='-fPIC', lddlflags='-shared -L/usr/local/lib -fstack-protector-strong' Characteristics of this binary (from libperl): Compile-time options: HAS_TIMES MULTIPLICITY PERLIO_LAYERS PERL_DONT_CREATE_GVSV PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_IMPLICIT_CONTEXT PERL_MALLOC_WRAP PERL_NEW_COPY_ON_WRITE PERL_PRESERVE_IVUV USE_64_BIT_ALL USE_64_BIT_INT USE_ITHREADS USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO USE_PERL_ATOF USE_REENTRANT_API . . . Built under linux Compiled at Mar 13 2016 11:54:18 @INC: /etc/perl /usr/local/lib/x86_64-linux-gnu/perl/5.22.1 /usr/local/share/perl/5.22.1 /usr/lib/x86_64-linux-gnu/perl5/5.22 /usr/share/perl5 /usr/lib/x86_64-linux-gnu/perl/5.22 /usr/share/perl/5.22 /usr/local/lib/site_perl /usr/lib/x86_64-linux-gnu/perl-base .
Here is another way that demonstrates the fix of modifying the @INC array similarly to how I described above: root@koha:~# cd /root root@koha:~# sudo -u library-koha perl -e 'use Authen::CAS::Client; print "Ok\n"' Can't locate Authen/CAS/Client/Response/Failure.pm: ./Authen/CAS/Client/Response/Failure.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106. BEGIN failed--compilation aborted at /usr/share/perl5/Authen/CAS/Client/Response.pm line 68. Compilation failed in require at /usr/share/perl5/Authen/CAS/Client.pm line 8. BEGIN failed--compilation aborted at /usr/share/perl5/Authen/CAS/Client.pm line 8. Compilation failed in require at -e line 1. BEGIN failed--compilation aborted at -e line 1. root@koha:~# sudo -u library-koha perl -e 'BEGIN {pop @INC} use Authen::CAS::Client; print "Ok\n"' Ok
I ran into this problem after upgrading from Koha 3.22, Ubuntu 14.04, Perl ? to Koha 17.05.05, Ubuntu 16.04.3 LTS, Perl v5.22.1 gather_print_notices.pl run from cron (sudo crontab -e) would fail with this error: Can't locate Koha/Exceptions/Exception.pm: ./Koha/Exceptions/Exception.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106. BEGIN failed--compilation aborted at (eval 14) line 3. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Exceptions.pm line 56. Compilation failed in require at /usr/share/koha/lib/Koha/DateUtils.pm line 22. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/DateUtils.pm line 22. Compilation failed in require at /usr/share/koha/lib/C4/Log.pm line 30. BEGIN failed--compilation aborted at /usr/share/koha/lib/C4/Log.pm line 30. Compilation failed in require at /usr/share/koha/lib/Koha/Config/SysPref.pm line 26. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Config/SysPref.pm line 26. Compilation failed in require at /usr/share/koha/lib/C4/Context.pm line 105. BEGIN failed--compilation aborted at /usr/share/koha/lib/C4/Context.pm line 105. Compilation failed in require at /usr/share/koha/bin/cronjobs/gather_print_notices.pl line 13. BEGIN failed--compilation aborted at /usr/share/koha/bin/cronjobs/gather_print_notices.pl line 13. Running the command in the crontab on the command line would succeed, no errors. Adding the "cd /tmp &&" trick in front of the commands in the crontab makes the errors go away.
We're getting the exact same error when trying to do a full rebuild from root cron, using /usr/sbin/koha-rebuild-zebra. Koha 17.05.05, Ubuntu 16.04.3 LTS, Perl v5.22.1 Cronjob: /usr/sbin/koha-rebuild-zebra -f -v SITE Error: Rebuilding authority indexes. Can't locate Koha/Exceptions/Exception.pm: ./Koha/Exceptions/Exception.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106. BEGIN failed--compilation aborted at (eval 15) line 3. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Exceptions.pm line 56. Compilation failed in require at /usr/share/koha/lib/Koha/DateUtils.pm line 22. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/DateUtils.pm line 22. Compilation failed in require at /usr/share/koha/lib/C4/Log.pm line 30. BEGIN failed--compilation aborted at /usr/share/koha/lib/C4/Log.pm line 30. Compilation failed in require at /usr/share/koha/lib/Koha/Config/SysPref.pm line 26. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Config/SysPref.pm line 26. Compilation failed in require at /usr/share/koha/lib/C4/Context.pm line 105. BEGIN failed--compilation aborted at /usr/share/koha/lib/C4/Context.pm line 105. Compilation failed in require at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 20. BEGIN failed--compilation aborted at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 20. Something went wrong rebuilding authority indexes for library Wed Dec 6 22:25:41 PST 2017 Rebuilding biblios indexes. Can't locate Koha/Exceptions/Exception.pm: ./Koha/Exceptions/Exception.pm: Permission denied at /usr/share/perl/5.22/base.pm line 97. ...propagated at /usr/share/perl/5.22/base.pm line 106. BEGIN failed--compilation aborted at (eval 15) line 3. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Exceptions.pm line 56. Compilation failed in require at /usr/share/koha/lib/Koha/DateUtils.pm line 22. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/DateUtils.pm line 22. Compilation failed in require at /usr/share/koha/lib/C4/Log.pm line 30. BEGIN failed--compilation aborted at /usr/share/koha/lib/C4/Log.pm line 30. Compilation failed in require at /usr/share/koha/lib/Koha/Config/SysPref.pm line 26. BEGIN failed--compilation aborted at /usr/share/koha/lib/Koha/Config/SysPref.pm line 26. Compilation failed in require at /usr/share/koha/lib/C4/Context.pm line 105. BEGIN failed--compilation aborted at /usr/share/koha/lib/C4/Context.pm line 105. Compilation failed in require at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 20. BEGIN failed--compilation aborted at /usr/share/koha/bin/migration_tools/rebuild_zebra.pl line 20. Something went wrong rebuilding biblio indexes for library The same work-around works: cd /tmp && /usr/sbin/koha-rebuild-zebra -f -v SITE
It looks like this one is not dead yet - Tomas, could you take another look as you hunted down the problem last time?
Understanding the problem is necessary to determine the best solution. The problem is this: - All cron jobs start in $HOME and those in /etc/cron* run as root which is /root. - Problems occur when using "sudo -u another_user" to demote from root to "another_user" which does not have read access to /root. - The @INC array is traversed when a module is to be loaded, but a recent update to Perl 5.22 (base.pm?) now results in a fatal error when it searches a directory that cannot be read. - The @INC array includes the current directory ('.') which is '/root' and unreadable by 'another_user', resulting in this error. - This error will also occur if run from the command line in a directory which is not readable my 'another_user'. Normally, crons are set up in the crontab file for the user under which it should run. Since it is rather unusual for a cron to sudo to another user, this is likely why this regression in Perl has gone largely unnoticed. Possible solutions: 1. Fix the regression in Perl 5.22. This is ideal and this may have already happened in more recent versions of Perl. However, many of us remain stuck with the version we have and we need a solution that will with with a broken Perl. Not a good short-term option. 2. Avoid using "sudo -u". While possibly ideal, it's convenient and simple for koha-foreach to be run as a system cron and switch to the user for each library. Therefore, this is not a good option. 3. Modify the @INC array to remove the '.' directory so that /root will never be searched. This could be done in a common location (e.g. /usr/share/koha/bin/kohalib.pl) as I suggested in Comment 40. This is risky because it assumes that no downstream module will ever depend upon '.' having been removed from @INC. While this solution has worked well for me, it probably is not ideal either. 4. Change the current directory to one that is readable by adding 'cd /tmp' to each cron job that performs a "sudo -u". This has become an established workaround, but requires modifying many cron entries. Also not an ideal solution. 5. Change the current directory to one that is readable by modifying all code that performs a "sudo -u". This may be as simple as adding "cd /tmp" at the beginning of /usr/sbin/koha-foreach and /usr/sbin/koha-rebuild-zebra. Possibly the best solution for now. Option #5 seems like the best one for now. It is a clear, one-line solution that doesn't require modifying any crons. Thoughts?
(In reply to Marco Moreno from comment #47) > Understanding the problem is necessary to determine the best solution. [...] > 5. Change the current directory to one that is readable by modifying all > code that performs a "sudo -u". This may be as simple as adding "cd /tmp" > at the beginning of /usr/sbin/koha-foreach and /usr/sbin/koha-rebuild-zebra. > Possibly the best solution for now. > > Option #5 seems like the best one for now. It is a clear, one-line solution > that doesn't require modifying any crons. I would agree. Put cd /tmp very close to the sudo -u. Adding it in cron is more confusing. Clear argumentation imo!
I can confirm this is still an issue on: Koha 17.11.01.000, Perl 5.22.1, Ubuntu 16.04.3 LTS Chris C's workaround fixes it for me.
Can anyone take care of this one?
(In reply to Jonathan Druart from comment #50) > Can anyone take care of this one? I'm taking care of it. I'm working on the same approach as bug 19546. Adding a --chdir option switch to koha-foreach.
(In reply to Marcel de Rooy from comment #48) > (In reply to Marco Moreno from comment #47) > > Understanding the problem is necessary to determine the best solution. > [...] > > 5. Change the current directory to one that is readable by modifying all > > code that performs a "sudo -u". This may be as simple as adding "cd /tmp" > > at the beginning of /usr/sbin/koha-foreach and /usr/sbin/koha-rebuild-zebra. > > Possibly the best solution for now. > > > > Option #5 seems like the best one for now. It is a clear, one-line solution > > that doesn't require modifying any crons. > > I would agree. Put cd /tmp very close to the sudo -u. Adding it in cron is > more confusing. > Clear argumentation imo! If there's a /tmp/Koha directory, without read permissions for the instance's user it will still fail. I think we should jump into the instance's home directory and I'm providing a patch to do so.
Created attachment 72368 [details] [review] Bug 17717: Add a --chdir option switch for koha-foreach Until Perl 5.26, the current directory is added to @INC when running a Perl script [1]. Having the current directory in @INC means it can be tried to be traversed when performing a lib lookup. Since version 5.18, Perl dies when it finds an unreadable directory (permissions) in @INC that needs to be traversed. This behaviour won't change because Perl devs consider it an enhancement to security. [2] Because of this, we need to make sure our scripts are ran **from** a directory in which they have read permissions. Ths patch adds a --chdir option switch to the **koha-foreach** wrapper script, that makes the inner shells/scripts to be ran within the Koha instance's user home directory. The change is trivial and should be QAed easily. I tested this on a prod server: - Create a /tmp/test.pl file containing: use Modern::Perl; use Cwd; my $dir = getcwd; warn $dir; 1; A) then create a cronjob entry to run it using koha-foreach: (in /etc/cron.d/test): 1/* * * * * root koha-foreach perl /tmp/test.pl - Once I noticed the cronjob ran, I used mutt to read the emails in the root user. => FAIL: ... Subject: Cron <root@koha> koha-foreach --enabled perl /tmp/test.pl "/root" "/root" "/root" "/root" "/root" ... B) I then used the patched koha-foreach with different results: => SUCCESS: ... Subject: Cron <root@koha> /root/koha-foreach --chdir --enabled perl /tmp/test.pl "/var/lib/koha/acaderc" "/var/lib/koha/agro" "/var/lib/koha/anc" "/var/lib/koha/arico" "/var/lib/koha/artes" ... So this patch's approach works. But... C) master's koha-foreach seems to work just the same... I think it is because of my previous attempt to fix this by using sudo in koha-shell. So I think environmental conditions affect the behaviour (which shell is configured for cron, sudo configuration, etc). ==== In conclusion, I think we should go ahead with this patch as it will solve peoples issues, and it is a right solution (option #5 on the list) to this Perl behaviour change. It doesn't cover other commands, but followup patches could do. I avoided /tmp as it is writable by any user... so it is an easy path for both exploiting by replacing some lib, and also because the existence of an unreadable dir that the interpreter could try to traverse (unreadable /tmp/Authen or /tmp/Koha will trigger the same error, and I assume people know what they are putting on the instance's dir, at least it will be easier to track). A followup patch takes care of making the cronjobs use --chdir when calling koha-foreach [1] https://lists.debian.org/debian-devel-announce/2016/08/msg00013.html [2] https://rt.perl.org/Public/Bug/Display.html?id=123795
Created attachment 72369 [details] [review] Bug 17717: Make cronjobs using koha-foreach use --chdir In order to patch production sites we need to adjust the shipped cronjobs so they are called with the --chdir option switch.
Created attachment 72372 [details] [review] Bug 17717: Add a --chdir option switch for koha-foreach Until Perl 5.26, the current directory is added to @INC when running a Perl script [1]. Having the current directory in @INC means it can be tried to be traversed when performing a lib lookup. Since version 5.18, Perl dies when it finds an unreadable directory (permissions) in @INC that needs to be traversed. This behaviour won't change because Perl devs consider it an enhancement to security. [2] Because of this, we need to make sure our scripts are ran **from** a directory in which they have read permissions. Ths patch adds a --chdir option switch to the **koha-foreach** wrapper script, that makes the inner shells/scripts to be ran within the Koha instance's user home directory. The change is trivial and should be QAed easily. I tested this on a prod server: - Create a /tmp/test.pl file containing: use Modern::Perl; use Cwd; my $dir = getcwd; warn $dir; 1; A) then create a cronjob entry to run it using koha-foreach: (in /etc/cron.d/test): 1/* * * * * root koha-foreach perl /tmp/test.pl - Once I noticed the cronjob ran, I used mutt to read the emails in the root user. => FAIL: ... Subject: Cron <root@koha> koha-foreach --enabled perl /tmp/test.pl "/root" "/root" "/root" "/root" "/root" ... B) I then used the patched koha-foreach with different results: => SUCCESS: ... Subject: Cron <root@koha> /root/koha-foreach --chdir --enabled perl /tmp/test.pl "/var/lib/koha/acaderc" "/var/lib/koha/agro" "/var/lib/koha/anc" "/var/lib/koha/arico" "/var/lib/koha/artes" ... So this patch's approach works. But... C) master's koha-foreach seems to work just the same... I think it is because of my previous attempt to fix this by using sudo in koha-shell. So I think environmental conditions affect the behaviour (which shell is configured for cron, sudo configuration, etc). ==== In conclusion, I think we should go ahead with this patch as it will solve peoples issues, and it is a right solution (option #5 on the list) to this Perl behaviour change. It doesn't cover other commands, but followup patches could do. I avoided /tmp as it is writable by any user... so it is an easy path for both exploiting by replacing some lib, and also because the existence of an unreadable dir that the interpreter could try to traverse (unreadable /tmp/Authen or /tmp/Koha will trigger the same error, and I assume people know what they are putting on the instance's dir, at least it will be easier to track). A followup patch takes care of making the cronjobs use --chdir when calling koha-foreach [1] https://lists.debian.org/debian-devel-announce/2016/08/msg00013.html [2] https://rt.perl.org/Public/Bug/Display.html?id=123795 Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 72373 [details] [review] Bug 17717: Make cronjobs using koha-foreach use --chdir In order to patch production sites we need to adjust the shipped cronjobs so they are called with the --chdir option switch. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Hmmm...I'm now reconsidering this and wonder if option #3 is really the best solution by removing '.' from @INC. You made a good point about /tmp being a concern. This, plus the fact that they have removed '.' from @INC in recent versions of Perl, has convinced me that having '.' in @INC is generally a very bad idea and a major security concern. Therefore, I want to propose revisiting comment #40 which removes '.' from @INC in a common library early in the bootstrapping process. This effectively undoes the "feature" added in Perl 5.18 and removed in Perl 5.26. Additionally, this prevents exploits that attempt to insert '.' via PERL5LIB. It is a single line of code, does nothing if '.' doesn't exist in @INC, and doesn't require modifying crons.
(In reply to Marco Moreno from comment #57) > Hmmm...I'm now reconsidering this and wonder if option #3 is really the best > solution by removing '.' from @INC. > > You made a good point about /tmp being a concern. This, plus the fact that > they have removed '.' from @INC in recent versions of Perl, has convinced me > that having '.' in @INC is generally a very bad idea and a major security > concern. > > Therefore, I want to propose revisiting comment #40 which removes '.' from > @INC in a common library early in the bootstrapping process. This > effectively undoes the "feature" added in Perl 5.18 and removed in Perl > 5.26. Additionally, this prevents exploits that attempt to insert '.' via > PERL5LIB. I considered that option. But we would need to do it on every script, and we would also be changing the current behaviour. Someone might be using their own lib (or version of some lib) on purpose and we'd be breaking them. So, for the general user base, we should stick with the cleaner solution IMHO.
Couldn't it be done in once in a common library that is used everywhere, like /usr/share/koha/bin/kohalib.pl? Sorry, I really don't know much about Koha's architecture and I trust you've studied this more than I have.
(In reply to Marco Moreno from comment #59) > Couldn't it be done in once in a common library that is used everywhere, > like /usr/share/koha/bin/kohalib.pl? > > Sorry, I really don't know much about Koha's architecture and I trust you've > studied this more than I have. I understand the kohalib.pl approach, but I think it is too hacky and I don't think we should use it at all. So a solution there I wouldn't agree. Also, not all CLI scripts written in perl use it...
Fair enough - trust your judgment.
Looking here. Something wrong still
Created attachment 72411 [details] [review] Bug 17717: Add a --chdir option switch for koha-foreach Until Perl 5.26, the current directory is added to @INC when running a Perl script [1]. Having the current directory in @INC means it can be tried to be traversed when performing a lib lookup. Since version 5.18, Perl dies when it finds an unreadable directory (permissions) in @INC that needs to be traversed. This behaviour won't change because Perl devs consider it an enhancement to security. [2] Because of this, we need to make sure our scripts are ran **from** a directory in which they have read permissions. Ths patch adds a --chdir option switch to the **koha-foreach** wrapper script, that makes the inner shells/scripts to be ran within the Koha instance's user home directory. The change is trivial and should be QAed easily. I tested this on a prod server: - Create a /tmp/test.pl file containing: use Modern::Perl; use Cwd; my $dir = getcwd; warn $dir; 1; A) then create a cronjob entry to run it using koha-foreach: (in /etc/cron.d/test): 1/* * * * * root koha-foreach perl /tmp/test.pl - Once I noticed the cronjob ran, I used mutt to read the emails in the root user. => FAIL: ... Subject: Cron <root@koha> koha-foreach --enabled perl /tmp/test.pl "/root" "/root" "/root" "/root" "/root" ... B) I then used the patched koha-foreach with different results: => SUCCESS: ... Subject: Cron <root@koha> /root/koha-foreach --chdir --enabled perl /tmp/test.pl "/var/lib/koha/acaderc" "/var/lib/koha/agro" "/var/lib/koha/anc" "/var/lib/koha/arico" "/var/lib/koha/artes" ... So this patch's approach works. But... C) master's koha-foreach seems to work just the same... I think it is because of my previous attempt to fix this by using sudo in koha-shell. So I think environmental conditions affect the behaviour (which shell is configured for cron, sudo configuration, etc). ==== In conclusion, I think we should go ahead with this patch as it will solve peoples issues, and it is a right solution (option #5 on the list) to this Perl behaviour change. It doesn't cover other commands, but followup patches could do. I avoided /tmp as it is writable by any user... so it is an easy path for both exploiting by replacing some lib, and also because the existence of an unreadable dir that the interpreter could try to traverse (unreadable /tmp/Authen or /tmp/Koha will trigger the same error, and I assume people know what they are putting on the instance's dir, at least it will be easier to track). A followup patch takes care of making the cronjobs use --chdir when calling koha-foreach [1] https://lists.debian.org/debian-devel-announce/2016/08/msg00013.html [2] https://rt.perl.org/Public/Bug/Display.html?id=123795 Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 72412 [details] [review] Bug 17717: Make cronjobs using koha-foreach use --chdir In order to patch production sites we need to adjust the shipped cronjobs so they are called with the --chdir option switch. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 72413 [details] [review] Bug 17717: (QA follow-up) Fix typo chdir This test does obviously not achieve the desired result: [ "chdir" != "no" ] Trivial fix. Adding the same quotes around starting_dir (just as for Bug 19546). Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Pushed to master for 18.05, thanks to all for your hard work.
Awesome work all, backported to stable for 17.11.05
Pushed to 17.05.x for v17.05.11
*** Bug 20786 has been marked as a duplicate of this bug. ***
Facing the same problem in my KOHA version 16.11. While trying to apply the said patch, I am not able to find the file 'koha-foreach.xml'. Guide me to find the file koha-foreach.xml so that may apply the patch. Regards, Dr. Nitesh Kumar Verma
Hi Nitesh, this bug has been fixed in 17.05 and later. 16.11 is no longer maintained and won't receive any updates. Please consider updating. If that's not an option asking on the mailing liston how to make this change locally would be better.