Bug 19546

Summary: Make koha-plack run Starman from the instance's directory
Product: Koha Reporter: Marcel de Rooy <m.de.rooy>
Component: REST APIAssignee: Tomás Cohen Arazi <tomascohen>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: major    
Priority: P5 - low CC: costalc, fridolin.somers, gmcharlt, jonathan.druart, mtj, nick, stefan.berndtsson, tomascohen
Version: 17.11   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17717
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 19546: Run starman from the instance's home dir
Bug 19546: Run starman from the instance's home dir
Bug 19546: Run starman from the instance's home dir
Bug 19546: (QA follow-up) Make the returning cd more fool proof
[17.11.x] Bug 19546: Run starman from the instance's home dir
[17.11.x] Bug 19546: (QA follow-up) Make the returning cd more fool proof

Description Marcel de Rooy 2017-10-30 09:25:40 UTC
See bug 19196 and my comment 40.
What happened? For just a short time I did not see the error. I merged the last changes on master, ran updatedatabase, restarted plack, flushed the cache and there it was again:

Error while loading /etc/koha/plack.psgi: Can't load application from file "api/v1/app.pl": Can't locate Mojolicious/Plugin/Koha/REST/Plugin/Pagination.pm:   ./Mojolicious/Plugin/Koha/REST/Plugin/Pagination.pm: Permission denied at (eval 1299) line 1.

This effectively breaks opac and staff client completely until I comment the change for the Pagination plugin again in Koha/REST/V1.pm.
Note that nothing special happened between Friday and now on this virtual Jessie box..
Comment 1 Marcel de Rooy 2017-10-30 09:31:21 UTC
Stopped/started apache, memcached, plack. To no effect.
Comment 2 Marcel de Rooy 2017-10-30 09:33:34 UTC
So just for the record:
I have a very simple workaround (but Tomas wont like it very much): Just comment the line until this path issue has been resolved completely.
Any other ideas ?
Comment 3 Tomás Cohen Arazi 2017-10-30 12:43:50 UTC
I never managed to reproduce it. Are you sure your setup is pointing to the right PERL5LIB?
Comment 4 Marcel de Rooy 2017-10-30 12:53:19 UTC
(In reply to Tomás Cohen Arazi from comment #3)
> I never managed to reproduce it. Are you sure your setup is pointing to the
> right PERL5LIB?

Ha. If it wasn't, I really should have had so much more problems..
And what about uncommenting a line in a library that is not used?
Comment 5 Marcel de Rooy 2017-10-31 14:02:35 UTC
(In reply to Marcel de Rooy from comment #4)
> (In reply to Tomás Cohen Arazi from comment #3)
> > I never managed to reproduce it. Are you sure your setup is pointing to the
> > right PERL5LIB?
> 
> Ha. If it wasn't, I really should have had so much more problems..
> And what about uncommenting a line in a library that is not used?

Just for the record:
Paths like PERL5LIB and KOHA_HOME are not the problem (verified).
It is something deeper in the install. Combination of libraries etc?

Since nobody doing a fresh install can confirm the problem, I will lower the severity of this bug first before closing it ;)
Picking minor for my dev box..
Comment 6 Jonathan Druart 2017-11-27 17:42:13 UTC
Closing, please reopen if it appears again.
Comment 7 Galen Charlton 2018-01-31 20:36:30 UTC
Reopening. Saw this happen on a Debian 8.10 system with a fresh install of 17.11.02 from package.

A (truly terrible workaround) got Plack running:

/usr/share/perl5/Mojolicious/Plugin && ln -s /usr/share/koha/lib/Koha Koha

Version of Mojolicious is 7.21 from the 7.21+dfsg-1~kohadev1 package.
Comment 8 Tomás Cohen Arazi 2018-02-02 19:41:47 UTC
(In reply to Galen Charlton from comment #7)
> Reopening. Saw this happen on a Debian 8.10 system with a fresh install of
> 17.11.02 from package.
> 
> A (truly terrible workaround) got Plack running:
> 
> /usr/share/perl5/Mojolicious/Plugin && ln -s /usr/share/koha/lib/Koha Koha
> 
> Version of Mojolicious is 7.21 from the 7.21+dfsg-1~kohadev1 package.

I don't manage to reproduce it.

I have installed 17.11.02-1 (17.11 repo) on a clean box:

   $ vagrant init debian/jessie64

only added port mappings to be able to access Koha (in Vagrantfile):
config.vm.network :forwarded_port, guest: 80,   host: 8080, auto_correct: true  # OPAC
config.vm.network :forwarded_port, guest: 8080, host: 8081, auto_correct: true  # INTRA

I then added 8080 to /etc/apache2/ports.conf and tweaked /etc/apache2/sites-enabled/kohadev.conf so intranet uses port 8080.

Once I populate the DB I manage to access the API (with and without Plack enabled). The only problem I noticed is that the rewrite bits for the API (in apache) are not working, so I needed to use /api/v1/app.pl/api/v1/patrons to access the API when Plack is disabled.

If you still have the VM in which you see the failure, could you try using a + sufix in the V1.pm line that includes the pagination plugin ? I'm not sure right now what was the way to avoid Mojolicious prefixing hte plugin name.
Comment 9 Marcel de Rooy 2018-02-02 21:19:25 UTC
I still have this problem on an older Jessie 8 box. A fresh Jessie install on another box does not have it. Looks to me that it is connected to the install history of various related dependencies to the Mojo stuff.
Comment 10 Tomás Cohen Arazi 2018-02-02 23:29:10 UTC
(In reply to Marcel de Rooy from comment #9)
> I still have this problem on an older Jessie 8 box. A fresh Jessie install
> on another box does not have it. Looks to me that it is connected to the
> install history of various related dependencies to the Mojo stuff.

Can you share that VM image? Send me a link by pm
Comment 11 Marcel de Rooy 2018-02-16 09:03:15 UTC
Well, this is interesting.
On my still reasonable fresh VM Jessie 8, this morning I got:

Error while loading /etc/koha/plack.psgi: Can't load application from file "/usr/share/koha/devclone/api/v1/app.pl": Can't locate Mojolicious/Plugin/Koha/REST/Plugin/Pagination.pm:   ./Mojolicious/Plugin/Koha/REST/Plugin/Pagination.pm: Permission denied at (eval 1511) line 1.

I am setting the Severity higher again. This really needs more attention from the Mojolicious guys!
Comment 12 Tomás Cohen Arazi 2018-02-16 12:05:00 UTC
(In reply to Marcel de Rooy from comment #11)
> Well, this is interesting.
> On my still reasonable fresh VM Jessie 8, this morning I got:
> 
> Error while loading /etc/koha/plack.psgi: Can't load application from file
> "/usr/share/koha/devclone/api/v1/app.pl": Can't locate
> Mojolicious/Plugin/Koha/REST/Plugin/Pagination.pm:  
> ./Mojolicious/Plugin/Koha/REST/Plugin/Pagination.pm: Permission denied at
> (eval 1511) line 1.
> 
> I am setting the Severity higher again. This really needs more attention
> from the Mojolicious guys!

Can you share your VM? I never managed to reproduce so there's nothing I can do right now!
Comment 13 Marcel de Rooy 2018-02-19 08:53:09 UTC
(In reply to Tomás Cohen Arazi from comment #12)
> Can you share your VM? I never managed to reproduce so there's nothing I can
> do right now!
Sorry, I can't. It is not mine :)
But it should not be that hard to reproduce since these are just gitified package installs on Jessie8.
One of the last things I did on the newest box was cpanminussing Parallel::ForkManager during QA. The module itself may not be the problem, and 
it may just be coincidental but this problem is about permissions and paths for perl libraries etc.
Comment 14 Tomás Cohen Arazi 2018-03-02 13:05:16 UTC
I managed to reproduce. This is the same issue introduced by Perl 5.18 we face with cronjobs. I'll provide a patch.
Comment 15 Tomás Cohen Arazi 2018-03-02 13:49:27 UTC
Created attachment 72365 [details] [review]
Bug 19546: Run starman from the instance's home dir

This patch makes koha-plack jump into the instance's home directory to
run.

It is required because Perl 5.18 introduced a breaking change that
makes perl die if @INC includes directories for which the user doesn't
have read permission, and need to be traversed when querying for a lib.
This is the case of '.', which is introduced automatically into @INC
until Perl 5.26 (which removes the 'feature').

The Mojolicious::Plugins lib prefixes the plugin names with
'Mojolicious::Plugin' so it first looks (for example) for
Mojolicious::Plugin::Koha::REST::Plugin::Pagination (Pagination is just
the first one on the list). When it looks for it at '.' it dies (because
of Perl's behaviour) so it doesn't query for the right namespace (the
following to try).

I only reproduced it in koha-testing-docker. To test, you just need to
try this patched koha-plack and make sure it doesn't break due to this
permissions issue.

To test:
- In your koha-testing-docker clone, run:
  $ docker-compose up -p test
- Open a shell inside the container (on a separate terminal):
  $ docker exec -it test_koha_1 bash
- From within the /root directory, restart plack:
  $ cd /root
  $ koha-plack --restart kohadev
=> FAIL: Logs show plack is broken due to permissions problems trying to
    find Mojolicious::Plugin::Koha::REST::Plugin::Pagination
- Use the patched script from this patch:
  $ /kohadevbox/koha/debian/scripts/koha-plack --restart kohadev
=> SUCCESS: Plack runs fine, no error in the logs
- Sign off :-D

Note: people who has environments in which the problems is reproducible,
    please test this version of koha-plack and stamp your sign-off,
    PLEASE.
Comment 16 Kyle M Hall 2018-03-02 15:53:04 UTC
Created attachment 72374 [details] [review]
Bug 19546: Run starman from the instance's home dir

This patch makes koha-plack jump into the instance's home directory to
run.

It is required because Perl 5.18 introduced a breaking change that
makes perl die if @INC includes directories for which the user doesn't
have read permission, and need to be traversed when querying for a lib.
This is the case of '.', which is introduced automatically into @INC
until Perl 5.26 (which removes the 'feature').

The Mojolicious::Plugins lib prefixes the plugin names with
'Mojolicious::Plugin' so it first looks (for example) for
Mojolicious::Plugin::Koha::REST::Plugin::Pagination (Pagination is just
the first one on the list). When it looks for it at '.' it dies (because
of Perl's behaviour) so it doesn't query for the right namespace (the
following to try).

I only reproduced it in koha-testing-docker. To test, you just need to
try this patched koha-plack and make sure it doesn't break due to this
permissions issue.

To test:
- In your koha-testing-docker clone, run:
  $ docker-compose up -p test
- Open a shell inside the container (on a separate terminal):
  $ docker exec -it test_koha_1 bash
- From within the /root directory, restart plack:
  $ cd /root
  $ koha-plack --restart kohadev
=> FAIL: Logs show plack is broken due to permissions problems trying to
    find Mojolicious::Plugin::Koha::REST::Plugin::Pagination
- Use the patched script from this patch:
  $ /kohadevbox/koha/debian/scripts/koha-plack --restart kohadev
=> SUCCESS: Plack runs fine, no error in the logs
- Sign off :-D

Note: people who has environments in which the problems is reproducible,
    please test this version of koha-plack and stamp your sign-off,
    PLEASE.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 17 Marcel de Rooy 2018-03-03 16:17:19 UTC
Great. Solution sounds plausible. Will test and QA. Just wondering why perl would build @INC different sometimes (recall that the problem was not completely constant). 
Glad that I am not the author of this reverted perl feature ;)
Comment 18 Marcel de Rooy 2018-03-05 09:31:21 UTC
Created attachment 72407 [details] [review]
Bug 19546: Run starman from the instance's home dir

This patch makes koha-plack jump into the instance's home directory to
run.

It is required because Perl 5.18 introduced a breaking change that
makes perl die if @INC includes directories for which the user doesn't
have read permission, and need to be traversed when querying for a lib.
This is the case of '.', which is introduced automatically into @INC
until Perl 5.26 (which removes the 'feature').

The Mojolicious::Plugins lib prefixes the plugin names with
'Mojolicious::Plugin' so it first looks (for example) for
Mojolicious::Plugin::Koha::REST::Plugin::Pagination (Pagination is just
the first one on the list). When it looks for it at '.' it dies (because
of Perl's behaviour) so it doesn't query for the right namespace (the
following to try).

I only reproduced it in koha-testing-docker. To test, you just need to
try this patched koha-plack and make sure it doesn't break due to this
permissions issue.

To test:
- In your koha-testing-docker clone, run:
  $ docker-compose up -p test
- Open a shell inside the container (on a separate terminal):
  $ docker exec -it test_koha_1 bash
- From within the /root directory, restart plack:
  $ cd /root
  $ koha-plack --restart kohadev
=> FAIL: Logs show plack is broken due to permissions problems trying to
    find Mojolicious::Plugin::Koha::REST::Plugin::Pagination
- Use the patched script from this patch:
  $ /kohadevbox/koha/debian/scripts/koha-plack --restart kohadev
=> SUCCESS: Plack runs fine, no error in the logs
- Sign off :-D

Note: people who has environments in which the problems is reproducible,
    please test this version of koha-plack and stamp your sign-off,
    PLEASE.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 19 Marcel de Rooy 2018-03-05 09:31:27 UTC
Created attachment 72408 [details] [review]
Bug 19546: (QA follow-up) Make the returning cd more fool proof

Trivial and hypothetical fix.
If you create /root/AA B (space in dirname), the quotes added here will
help you to get back where you came from.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 20 Nick Clemens 2018-03-15 09:21:35 UTC
Pushed to master for 18.05, thanks to all for your hard work.
Comment 21 Nick Clemens 2018-04-06 17:13:46 UTC
Does not apply cleanly to 17.11 - let me know and provide a rebase or I can try
Comment 22 Tomás Cohen Arazi 2018-04-06 17:58:18 UTC
Created attachment 73794 [details] [review]
[17.11.x] Bug 19546: Run starman from the instance's home dir

This patch makes koha-plack jump into the instance's home directory to
run.

It is required because Perl 5.18 introduced a breaking change that
makes perl die if @INC includes directories for which the user doesn't
have read permission, and need to be traversed when querying for a lib.
This is the case of '.', which is introduced automatically into @INC
until Perl 5.26 (which removes the 'feature').

The Mojolicious::Plugins lib prefixes the plugin names with
'Mojolicious::Plugin' so it first looks (for example) for
Mojolicious::Plugin::Koha::REST::Plugin::Pagination (Pagination is just
the first one on the list). When it looks for it at '.' it dies (because
of Perl's behaviour) so it doesn't query for the right namespace (the
following to try).

I only reproduced it in koha-testing-docker. To test, you just need to
try this patched koha-plack and make sure it doesn't break due to this
permissions issue.

To test:
- In your koha-testing-docker clone, run:
  $ docker-compose up -p test
- Open a shell inside the container (on a separate terminal):
  $ docker exec -it test_koha_1 bash
- From within the /root directory, restart plack:
  $ cd /root
  $ koha-plack --restart kohadev
=> FAIL: Logs show plack is broken due to permissions problems trying to
    find Mojolicious::Plugin::Koha::REST::Plugin::Pagination
- Use the patched script from this patch:
  $ /kohadevbox/koha/debian/scripts/koha-plack --restart kohadev
=> SUCCESS: Plack runs fine, no error in the logs
- Sign off :-D

Note: people who has environments in which the problems is reproducible,
    please test this version of koha-plack and stamp your sign-off,
    PLEASE.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 23 Tomás Cohen Arazi 2018-04-06 17:58:30 UTC
Created attachment 73795 [details] [review]
[17.11.x] Bug 19546: (QA follow-up) Make the returning cd more fool proof

Trivial and hypothetical fix.
If you create /root/AA B (space in dirname), the quotes added here will
help you to get back where you came from.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 24 Tomás Cohen Arazi 2018-04-06 18:00:13 UTC
I removed the dependency on bu 19196 because it only highlighted the underlying problem.
Comment 25 Nick Clemens 2018-05-22 11:05:48 UTC
Pushed to stable for 17.11.X - had to change the variable from $instance-user to $instancename-koha 

http://git.koha-community.org/gitweb/?p=koha.git;a=commit;h=7963834a937909c59060dec61f8bd930f50af6b0
Comment 26 Fridolin Somers 2018-05-23 05:00:53 UTC
I this also for 17.05.x ?