Bug 25245 - Add a plugin hook to allow running code on a nightly basis
Summary: Add a plugin hook to allow running code on a nightly basis
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Plugin architecture (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Martin Renvoize (ashimema)
QA Contact: Tomás Cohen Arazi (tcohen)
URL:
Keywords:
: 20897 (view as bug list)
Depends on:
Blocks: 27820
  Show dependency treegraph
 
Reported: 2020-04-22 10:57 UTC by Martin Renvoize (ashimema)
Modified: 2021-12-13 21:11 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This patchset adds a new cronjob script to Koha, plugins_nightly.pl This script will check for plugins that have registered a cronjob_nightly method and execute that method. This enhancement allows users to install and setup plugins that require cronjobs without backend system changes and prevents the addition of new cronjob files for each plugin.
Version(s) released in:
21.05.00,20.11.03
Circulation function:


Attachments
Bug 25245: Add plugins_nightly.pl cronjob script (2.00 KB, patch)
2020-04-22 11:05 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl cronjob script (2.00 KB, patch)
2020-04-22 11:12 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.00 KB, patch)
2020-04-22 11:12 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl cronjob script (2.07 KB, patch)
2020-04-22 17:02 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.07 KB, patch)
2020-04-22 17:03 UTC, Kyle M Hall (khall)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl cronjob script (1.95 KB, patch)
2020-04-27 12:32 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: (follow-up) Use Koha::Logger (1.03 KB, patch)
2020-04-27 12:32 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.00 KB, patch)
2020-04-27 12:32 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl cronjob script (1.95 KB, patch)
2020-04-27 12:45 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: (follow-up) Use Koha::Logger (995 bytes, patch)
2020-04-27 12:45 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.00 KB, patch)
2020-04-27 12:45 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.46 KB, patch)
2020-04-27 12:49 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25188: Make circulation notes more prominent on the patron details tab (2.01 KB, patch)
2020-05-05 06:53 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl cronjob script (2.00 KB, patch)
2020-05-27 14:37 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 25245: (follow-up) Use Koha::Logger (1.03 KB, patch)
2020-05-27 14:37 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.51 KB, patch)
2020-05-27 14:37 UTC, Tomás Cohen Arazi (tcohen)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl cronjob script (2.05 KB, patch)
2020-06-19 11:21 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: (follow-up) Use Koha::Logger (1.08 KB, patch)
2020-06-19 11:22 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 25245: Add plugins_nightly.pl to the default cron (1.56 KB, patch)
2020-06-19 11:22 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Renvoize (ashimema) 2020-04-22 10:57:45 UTC
Some plugins, Recalls for example, require the addition of a cronjob that runs on a nightly basis.

We could greatly ease the installation of such plugins by introducing a new default cron script to that runs nightly and simply iterated through plugins implementing a cronjob_nightly method.
Comment 1 Martin Renvoize (ashimema) 2020-04-22 11:05:46 UTC
Created attachment 103454 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.
Comment 2 Martin Renvoize (ashimema) 2020-04-22 11:12:34 UTC
Created attachment 103460 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.
Comment 3 Martin Renvoize (ashimema) 2020-04-22 11:12:37 UTC
Created attachment 103461 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron
Comment 4 Martin Renvoize (ashimema) 2020-04-22 11:19:28 UTC
Test plan

1/ Apply patch
2/ Ensure your cron setup calls the new plugins_nightly.pl
3/ Install the recalls plugin and skip the 'setup cron_nightly.pl' step
4/ The recalls plugin should function correctly even though you've not done the manual cron intervention.
5/ Signoff
Comment 5 Kyle M Hall (khall) 2020-04-22 17:02:19 UTC
Created attachment 103497 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 6 Kyle M Hall (khall) 2020-04-22 17:03:03 UTC
Created attachment 103498 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 7 Marcel de Rooy 2020-04-24 06:22:25 UTC
Some QA comments:
use Modern::Perl
Where do you need FindBin for here?
I understand the test on enable_plugins but I feel that it should be part of GetPlugins somehow. If you did not enable them, GetPlugins should take care of it and return nothing.
The if(@plugins) is not needed. You are looping thru @plugins inside.
Why do you warn $_ ?

Second patch
If you are adjusting the debian koha-conf, you should also touch the other one.
Comment 8 Martin Renvoize (ashimema) 2020-04-27 12:32:45 UTC
Created attachment 103767 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.
Comment 9 Martin Renvoize (ashimema) 2020-04-27 12:32:48 UTC
Created attachment 103768 [details] [review]
Bug 25245: (follow-up) Use Koha::Logger

Update script to use Koha::Logger to capture method failures.
Comment 10 Martin Renvoize (ashimema) 2020-04-27 12:32:51 UTC
Created attachment 103769 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron
Comment 11 Martin Renvoize (ashimema) 2020-04-27 12:45:06 UTC
Created attachment 103774 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.
Comment 12 Martin Renvoize (ashimema) 2020-04-27 12:45:09 UTC
Created attachment 103775 [details] [review]
Bug 25245: (follow-up) Use Koha::Logger

Update script to use Koha::Logger to capture method failures.
Comment 13 Martin Renvoize (ashimema) 2020-04-27 12:45:13 UTC
Created attachment 103776 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron
Comment 14 Martin Renvoize (ashimema) 2020-04-27 12:49:05 UTC
Created attachment 103778 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron
Comment 15 Martin Renvoize (ashimema) 2020-04-27 12:52:09 UTC
(In reply to Marcel de Rooy from comment #7)
> Some QA comments:
> use Modern::Perl

Corrected

> Where do you need FindBin for here?

We don't, I've removed it.. I coded it in a hurry whilst I thought of the idea ;)

> I understand the test on enable_plugins but I feel that it should be part of
> GetPlugins somehow. If you did not enable them, GetPlugins should take care
> of it and return nothing.

Agreed, but I'm not sure of the other implications.. as such I'd prefer to do that in a distinct bug.

> The if(@plugins) is not needed. You are looping thru @plugins inside.

Agreed, corrected.

> Why do you warn $_ ?

Following prior art. It makes sense to warn like this for such errors.  I have however added a Koha::Logger based log line too now so we don't lose warnings for cases where cron sends them to dev/null.

> 
> Second patch
> If you are adjusting the debian koha-conf, you should also touch the other
> one.

Amended patch.
Comment 16 David Cook 2020-04-28 00:53:10 UTC
Oh... I like the sound of this. Could be very handy.
Comment 17 Marcel de Rooy 2020-04-28 13:07:25 UTC
(In reply to Martin Renvoize from comment #15)
> > Why do you warn $_ ?
> 
> Following prior art. It makes sense to warn like this for such errors.  I
> have however added a Koha::Logger based log line too now so we don't lose
> warnings for cases where cron sends them to dev/null.

You probably mean something else? Like $@ ? The wonderous names of Perl special variables. I am really not sure what $_ would be referring to in this catch block?
Comment 18 Tomás Cohen Arazi (tcohen) 2020-04-28 13:10:24 UTC
(In reply to Marcel de Rooy from comment #17)
> (In reply to Martin Renvoize from comment #15)
> > > Why do you warn $_ ?
> > 
> > Following prior art. It makes sense to warn like this for such errors.  I
> > have however added a Koha::Logger based log line too now so we don't lose
> > warnings for cases where cron sends them to dev/null.
> 
> You probably mean something else? Like $@ ? The wonderous names of Perl
> special variables. I am really not sure what $_ would be referring to in
> this catch block?

eval sets $@, Try::Tiny preserves it just in case you need it for other purposes, and sets $_. So, yes: it is the correct way with Try::Tiny. Look at all the API controllers for examples.
Comment 19 Marcel de Rooy 2020-04-28 13:13:04 UTC
(In reply to Tomás Cohen Arazi from comment #18)
> (In reply to Marcel de Rooy from comment #17)
> > (In reply to Martin Renvoize from comment #15)
> > > > Why do you warn $_ ?
> > > 
> > > Following prior art. It makes sense to warn like this for such errors.  I
> > > have however added a Koha::Logger based log line too now so we don't lose
> > > warnings for cases where cron sends them to dev/null.
> > 
> > You probably mean something else? Like $@ ? The wonderous names of Perl
> > special variables. I am really not sure what $_ would be referring to in
> > this catch block?
> 
> eval sets $@, Try::Tiny preserves it just in case you need it for other
> purposes, and sets $_. So, yes: it is the correct way with Try::Tiny. Look
> at all the API controllers for examples.

Ah great Thanks
Comment 20 Martin Renvoize (ashimema) 2020-04-28 16:05:01 UTC
Ah.. sorry.. I hadn't spotted you were worried about the$_ as apposed to $@.. I thought it was the introduction of a 'warn' at all you were commenting on.

Yes, as Tomas explains, Try::Tiny catches $@ and throws it into $_.. it's a peculiarity of that module.. I've wondered for a while about switching it out with Syntax::Keyword::Try which I believe is now recommended by the Perl echelons.
Comment 21 Jonathan Druart 2020-05-04 10:37:47 UTC
I have the feeling that this should have been implemented on top of bug 22417 (with bug 1993 in mind).
Comment 22 Kyle M Hall (khall) 2020-05-04 13:13:47 UTC
(In reply to Jonathan Druart from comment #21)
> I have the feeling that this should have been implemented on top of bug
> 22417 (with bug 1993 in mind).

Is there any reason this cannot be updated to use the functionality of those bugs once they've been taken care of? It doesn't look like there is a definite end in site for them.
Comment 23 Tomás Cohen Arazi (tcohen) 2020-05-04 13:23:01 UTC
(In reply to Kyle M Hall from comment #22)
> (In reply to Jonathan Druart from comment #21)
> > I have the feeling that this should have been implemented on top of bug
> > 22417 (with bug 1993 in mind).
> 
> Is there any reason this cannot be updated to use the functionality of those
> bugs once they've been taken care of? It doesn't look like there is a
> definite end in site for them.

I agree here. Bug 22417 is a longer shot IMHO.
Comment 24 Jonathan Druart 2020-05-04 13:46:45 UTC
The main inconvenient of this implementation is that the time the job will start is not configurable.
Comment 25 David Cook 2020-05-04 23:53:43 UTC
(In reply to Jonathan Druart from comment #24)
> The main inconvenient of this implementation is that the time the job will
> start is not configurable.

That's true although that's a criticism I have of many of the daily cronjobs. (We're switching over more to using the Koha Debian packages, and noticed one of the gaps between it and our custom installs is the inflexibility of the cronjobs out of the box.)
Comment 26 David Cook 2020-05-05 00:06:55 UTC
(In reply to Tomás Cohen Arazi from comment #23)
> (In reply to Kyle M Hall from comment #22)
> > (In reply to Jonathan Druart from comment #21)
> > > I have the feeling that this should have been implemented on top of bug
> > > 22417 (with bug 1993 in mind).
> > 
> > Is there any reason this cannot be updated to use the functionality of those
> > bugs once they've been taken care of? It doesn't look like there is a
> > definite end in site for them.
> 
> I agree here. Bug 22417 is a longer shot IMHO.

I agree with everyone in a way.

I've spent years talking about and working on task scheduler functionality for Koha, and I don't think it's happening any time soon. Cron is a well-established familiar task scheduler that can do a good enough job. 

I'd argue Bug 22417 tries to be too comprehensive, and that we'd be better off just adding RabbitMQ and Net::Stomp as dependencies, and then doing a light experimental implementation here. However, even if we had only 1 worker consuming from RabbitMQ, that's another process to manage. It adds new administrative overhead and fragility that we don't have by continuing to use cron.

Frido also made a good point back in Portland about queue based task schedulers (like the one I wrote). From a vendor point of view, you might not want all your Koha instances running their background jobs at the same time. If those background jobs use APIs, you might end up hitting rate limits for instance. Or if you're sending out lots of emails, you might get flagged as a spammer. So it becomes prudent for vendors to be able to stagger background jobs on a server.

Of course, that criticism applies to using cron.daily for background jobs too.
Comment 27 David Cook 2020-05-05 01:00:21 UTC
(In reply to David Cook from comment #26)
> I'd argue Bug 22417 tries to be too comprehensive, and that we'd be better
> off just adding RabbitMQ and Net::Stomp as dependencies, and then doing a
> light experimental implementation here.

Actually, that wouldn't work. We would need a proper task scheduler.

It's a nice idea in theory, but I don't think it's happening any time soon, so I'd say go with cron and try to solve this one in the future :/.
Comment 28 David Cook 2020-05-05 03:10:39 UTC
Actually, in Jonathan's defence again, on Ubuntu 18.04 it wouldn't be a "nightly" method so much as a daily/every morning method when run in cron.daily, as it's set up to run at 6:25am out of the box.

Maybe it should go in /etc/cron.d/koha-common instead with a better time defined. And then local admins can customize that.
Comment 29 Katrin Fischer 2020-05-05 06:46:27 UTC
I feel like this could already help a lot of plugin authors and libraries as it will allow to streamline installing plugins that need to process data overnight. An even more flexible solution would be great (agreed!), but maybe this is a good enough first step even with its limitations?
Comment 30 Katrin Fischer 2020-05-05 06:53:49 UTC
Created attachment 104345 [details] [review]
Bug 25188: Make circulation notes more prominent on the patron details tab

This patch adds a more prominent circulation note to the moremember.pl
details screen.

To test:

1) Add a circulation note to a patron record.
2) Note that it displays prominently on the checkout tab, but only under
the Library Use block on the details tab.
3) Apply the patch.
4) Check that the note is now displayed prominently at the top of the
details (moremember.pl) screen.

https://bugs.koha-community.org/show_bug.cgi?id=25118

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 31 Katrin Fischer 2020-05-05 06:54:38 UTC
Comment on attachment 104345 [details] [review]
Bug 25188: Make circulation notes more prominent on the patron details tab

Wrong bug number...
Comment 32 Martin Renvoize (ashimema) 2020-05-05 07:14:22 UTC
I didn't want to get stuck in the queue waiting for any of the schedulers rewrite bugs so it was a deliberate choice not to base this code upon them.  Cron is a simple system and this is really designed for simple use cases... the idea being we just want a task to run each 'night', but we're not too fussy as to when.

We already have a hook added to trigger plugin actions prior to the fines job and I like the idea of slowly adding such triggers to the start/end of other regular tasks so one can dovetail workloads as required.. but for general 'process stuff daily' task I personally feel this is enough.

You can happily gain a little more control over the scheduling within your plugin code by doing a date comparison as the first part of the job in your plugin so one could for instance set the tasks to take place ever X days rather than nightly, or every Friday for example.

It would also allow simple installation of the existing recalls plugin which currently requires lots of background manipulation to install.
Comment 33 David Cook 2020-05-05 23:11:10 UTC
(In reply to Martin Renvoize from comment #32)
> You can happily gain a little more control over the scheduling within your
> plugin code by doing a date comparison as the first part of the job in your
> plugin so one could for instance set the tasks to take place ever X days
> rather than nightly, or every Friday for example.
> 

What do you think about putting the cronjob in /etc/cron.d/koha-common instead of /etc/cron.daily/koha-common?

The former could run at very configurable time whereas the latter runs at 6:25am out of the box. 

I could see Koha sysadmins customizing /etc/cron.d/koha-common much more readily than /etc/crontab (as you couldn't want to changethe time for all daily cronjobs).

I think that would be a very reasonable change to make to the current patch set?
Comment 34 Katrin Fischer 2020-05-18 10:18:58 UTC
> What do you think about putting the cronjob in /etc/cron.d/koha-common
> instead of /etc/cron.daily/koha-common?
> 
> The former could run at very configurable time whereas the latter runs at
> 6:25am out of the box. 
> 
> I could see Koha sysadmins customizing /etc/cron.d/koha-common much more
> readily than /etc/crontab (as you couldn't want to changethe time for all
> daily cronjobs).
> 
> I think that would be a very reasonable change to make to the current patch
> set?

A nightly job in the daily file makes sense to me - that's where I would be looking for it. And I'd rather schedule it at a standard time tha a new arbitrary one. And if you want another time, you could just move it?
Comment 35 Martin Renvoize (ashimema) 2020-05-18 11:02:02 UTC
I agree with Katrin here I think.. I would say it's clearer to have the default in the .daily file as it's intended to be run daily and for an admin to move it to .d if they are keen to specify an exact time.

Also, as an aside, it is my understanding that these two schemes work in different ways.. cron.d runs more like the original cron and just 'does a thing at time X' where as cron.daily (and friends) uses anacron in the background and as such 'does thing some time after X, with random delay Y and retries if the server is down'.
Comment 36 David Cook 2020-05-19 02:20:37 UTC
(In reply to Katrin Fischer from comment #34)
> A nightly job in the daily file makes sense to me - that's where I would be
> looking for it. And I'd rather schedule it at a standard time tha a new
> arbitrary one. And if you want another time, you could just move it?

That puts the onus on the system administrator for every out-of-the-box install to change it though from 6:30am to a different time.
Comment 37 David Cook 2020-05-19 02:26:29 UTC
(In reply to Martin Renvoize from comment #35)
> I agree with Katrin here I think.. I would say it's clearer to have the
> default in the .daily file as it's intended to be run daily and for an admin
> to move it to .d if they are keen to specify an exact time.
> 

I suppose so. I'm just thinking about out-of-the-box minimal installations. It's no drama for me to change a cronjob time, but I wonder a bit about many small scale sysadmins out there using Koha. 

> Also, as an aside, it is my understanding that these two schemes work in
> different ways.. cron.d runs more like the original cron and just 'does a
> thing at time X' where as cron.daily (and friends) uses anacron in the
> background and as such 'does thing some time after X, with random delay Y
> and retries if the server is down'.

anacron should never be installed on a server. It's really more suited to desktops and laptops. 

That said, /etc/crontab will use it if it's present and executable.

I'm looking at a customized Debian 9 Jessie and I see anacron installed and executable, and I'm looking at a Ubuntu 18.04 Bionic and I see that anacron is not installed. 

Sure enough, the Debian Jessie appears to be a Desktop image, while the Ubuntu is a headless server image. 

I think that actually helps make my case a bit for not using /etc/cron.daily heh.

But that's just my 2 cents. It's not a hill I want to die on by any means. Happy for it to go in /etc/cron.daily if everyone else does.
Comment 38 Tomás Cohen Arazi (tcohen) 2020-05-27 14:37:19 UTC
Created attachment 105400 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 39 Tomás Cohen Arazi (tcohen) 2020-05-27 14:37:25 UTC
Created attachment 105401 [details] [review]
Bug 25245: (follow-up) Use Koha::Logger

Update script to use Koha::Logger to capture method failures.

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 40 Tomás Cohen Arazi (tcohen) 2020-05-27 14:37:30 UTC
Created attachment 105402 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 41 Tomás Cohen Arazi (tcohen) 2020-05-27 14:42:59 UTC
I think this implementation is simple and covers the use case it is intended to. If we happen to have a task scheduler in a future, we could replace this and many other things we do with cron with it.

I don't think the Koha::Logger follow-up should be pushed along with this, as we don't have a 'cron' entry in log4perl.
Comment 42 David Cook 2020-05-27 23:23:53 UTC
(In reply to Tomás Cohen Arazi from comment #41)
> I think this implementation is simple and covers the use case it is intended
> to. If we happen to have a task scheduler in a future, we could replace this
> and many other things we do with cron with it.
> 

Agreed.

With one of my OAI-PMH plugins, I'm writing a custom task scheduler, but it's a complex piece of work, so I'd be happy to use the plugin nightly hook in the meantime (especially if it were backported to 19.11).
Comment 43 Jonathan Druart 2020-06-15 08:52:10 UTC
I am seeing only 1 signed-off-by lines per patches.
Comment 44 Martin Renvoize (ashimema) 2020-06-19 11:21:56 UTC
Created attachment 106067 [details] [review]
Bug 25245: Add plugins_nightly.pl cronjob script

This script simply iterates through installed plugins that impliment a
cronjob_nightly method and runs said method.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 45 Martin Renvoize (ashimema) 2020-06-19 11:22:00 UTC
Created attachment 106068 [details] [review]
Bug 25245: (follow-up) Use Koha::Logger

Update script to use Koha::Logger to capture method failures.

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 46 Martin Renvoize (ashimema) 2020-06-19 11:22:04 UTC
Created attachment 106069 [details] [review]
Bug 25245: Add plugins_nightly.pl to the default cron

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 47 Martin Renvoize (ashimema) 2020-06-19 11:22:32 UTC
Restored SO lines.
Comment 48 Martin Renvoize (ashimema) 2020-07-02 12:42:15 UTC
*** Bug 20897 has been marked as a duplicate of this bug. ***
Comment 49 Jonathan Druart 2020-10-09 10:14:05 UTC
(In reply to Jonathan Druart from comment #21)
> I have the feeling that this should have been implemented on top of bug
> 22417 (with bug 1993 in mind).

Now that we have bug 22417 into master, do we still need this?
Comment 50 Martin Renvoize (ashimema) 2020-11-30 14:17:43 UTC
Could you possibly write an example of how you'd expect a plugin to use the new task queue stuff as an alternative to this?  I'm happy for this bug to be replaced with such an example.. right now I don't know enough of how that code fits together to really do such an example myself.
Comment 51 David Cook 2020-12-01 04:41:05 UTC
(In reply to Martin Renvoize from comment #50)
> Could you possibly write an example of how you'd expect a plugin to use the
> new task queue stuff as an alternative to this?  I'm happy for this bug to
> be replaced with such an example.. right now I don't know enough of how that
> code fits together to really do such an example myself.

I don't think it's possible at the moment, but here's what I would envision:

1. Have a module/function registry for koha-worker to check when it gets a message
2. A plugin registers a module/function when it's installed
3. The plugin provides an API where it gives an immediate response and stages some data but does the actual processing via the task queue (ie koha-worker)
4. The koha-worker verifies the message against the registry, and invokes the code provided by the Koha plugin

I think that Tomas may have had a similar idea?
Comment 52 Jonathan Druart 2020-12-01 08:31:13 UTC
An easy (but lazy) implementation could be:
- Have a Koha::BackgroundJob::Plugin
- Your plugin class inherit it and has a job_type="plugin|Your::Plugin::Class"
- Your plugin implement ->process

You can now call ->enqueue to enqueue the job, and the worker will process it when possible.

However we don't have the "at" part here, it should be done on a separate bug report with bug 1993 in mind.

I think Tomas's idea was to have REST API routes to enqueue new jobs.
Comment 53 David Cook 2020-12-01 23:27:41 UTC
(In reply to Jonathan Druart from comment #52)
> An easy (but lazy) implementation could be:
> - Have a Koha::BackgroundJob::Plugin
> - Your plugin class inherit it and has a
> job_type="plugin|Your::Plugin::Class"
> - Your plugin implement ->process
> 
> You can now call ->enqueue to enqueue the job, and the worker will process
> it when possible.
 
Could you elaborate on this part? The koha-worker code would need to be updated to support that job_type="plugin|Your::Plugin::Class", right?

> However we don't have the "at" part here, it should be done on a separate
> bug report with bug 1993 in mind.
> 

I'd love to do a POC for this. I'll comment on Bug 1993. (Now to find the time/energy...)

> I think Tomas's idea was to have REST API routes to enqueue new jobs.

+1 

I like the idea although we'd want that API route to be private/highly privileged. Otherwise it might be accidentally abused. (I have been thinking how Koha could use a better separation of private/public API routes...)
Comment 54 Martin Renvoize (ashimema) 2020-12-09 16:42:00 UTC
Firstly, this bug was all about adding the ability to just run a task added by a plugin at regular intervals.. it was added to resolve the need to set up all sorts of additional cron jobs for the various plugins out in the wild that document 'when you install this plugin you also need to go on the server and add x, y and z cron tasks'.  Yes, a task scheduler would be great, but I really didn't want to wait another 10 years whilst people made up their mind on how to implement one.

I wasn't even attempting to resolve the need for adding tasks to the FIFO task queue... to me, that's a different process and bug entirely...

Task Schedular != Task Queue

As for the task queue, now I've looked at it.. I'm rather disappointed that it's hardcoded to a list of tasks, doesn't allow for parallelism, doesn't offer retries or failure states, doesn't even allow for multiple workers?  Perhaps I was spoilt with Minion in previous projects.. personally I would have loved to have simply adopted it.. I thought the biggest thing against it was that it recommends the use or Mojo::Pg as the backend and we use MySQL.... what's the difference in requiring people to install RabbitMQ vs requiring them to install Postgres?... there is also a well maintained MySQL backend that implements a clever form of PubSub to get around the fact that MySQL doesn't do pubsub natively.


As for a simpler scheduler.. why not just keep a list of tasks and execution times akin to cron in a table and query it every minute.. the exact same way cron does... and leave FIFO tasks to the queue implementation we've now got.
Comment 55 David Cook 2020-12-09 22:31:54 UTC
(In reply to Martin Renvoize from comment #54)
> Yes, a task scheduler would be
> great, but I really didn't want to wait another 10 years whilst people made
> up their mind on how to implement one.
 
Very fair point.

> 
> Task Schedular != Task Queue
> 

Also very true.

> As for the task queue, now I've looked at it.. I'm rather disappointed that
> it's hardcoded to a list of tasks, doesn't allow for parallelism, doesn't
> offer retries or failure states, doesn't even allow for multiple workers? 

I'm disappointed by the same things, but no one (myself included) was willing to work on those things. That said, it shouldn't be too hard to refactor (brand new code...) to have a more dynamic list of potential tasks and using multiple workers would be fairly trivial to add too. I haven't looked at the code recently enough to remember how it handles retries/failures, but that's something that would be fairly app specific anyway I think, so something that can certainly grow over time.

These are changes I want to make at some point, but I don't have the resources to do them. 

> Perhaps I was spoilt with Minion in previous projects.. personally I would
> have loved to have simply adopted it.. 

I wasn't super fond of Minion when it was first mentioned, although recently I was re-reading the documentation and it is more full featured than I originally thought. It probably would have been sufficient. In hindsight, we probably should have gone that route. Considering the "sunk cost fallacy", perhaps we should still go that way. Even in terms of sunk cost, we haven't actually invested too much in RabbitMQ yet.

(Regarding RabbitMQ, I like how we can use a standard messaging protocol, which gives us language agnosticism - it would be nice to use more task-appropriate languages rather than just using Perl for everything. I like how it's scalable, although few Koha instances need to scale. These days, you can use a managed RabbitMQ via AWS, so you don't even have to maintain that infrastructure, if you have the budget.)

> I thought the biggest thing against
> it was that it recommends the use or Mojo::Pg as the backend and we use
> MySQL.... 

I think that was one of the "cons", but when it was first mentioned I also don't think that Koha had really adopted Mojolicious as fully as it has now. I think there was still a fair bit of skepticism about Mojolicious at that point in time.

> what's the difference in requiring people to install RabbitMQ vs
> requiring them to install Postgres?... 
> there is also a well maintained MySQL
> backend that implements a clever form of PubSub to get around the fact that
> MySQL doesn't do pubsub natively.
> 

I think it's a bit absurd to require 2 different relational databases on the face of it, but that's a fair point from a technical perspective. I've noticed the MySQL backend as well and it seems reasonable enough at a glance.

> As for a simpler scheduler.. why not just keep a list of tasks and execution
> times akin to cron in a table and query it every minute.. the exact same way
> cron does... and leave FIFO tasks to the queue implementation we've now got.

Many years ago, Galen and I talked about keeping a list of tasks and execution times in a table and having a cronjob run every minute to examine the times and run the tasks, but no one got around to doing it. Personally, I'm more ambitious than that now and have had use cases that required scheduling granularity of less than 1/minute. However, my ambitions certainly outstrip the time/energy that I have available. Even though I have the technical ability, I don't have the logistical resources to get it done in a timely way. 

Now I'm just being pedantic, but the simple scheduler proposal is different to cron. If I recall correctly, cron keeps all the tasks in memory, and it only does I/O when it detects a change to a crontab. But polling a database every 1/minute isn't a drama. It's a scheduler that wouldn't work for some of my use cases, but it would meet the majority of Koha's needs, so it's probably the way to go.
Comment 56 David Cook 2020-12-09 22:34:52 UTC
Oh, another point about not using cron-based schedulers. Cron is a royal pain when you're working with containers. It would nice not to be continually wedded to a 1970s technology...
Comment 57 David Cook 2020-12-09 23:09:59 UTC
Regarding my above comments, while I have ambitions for a bespoke task scheduler, Martin is correct that it is unlikely that I am going to write such a task scheduler any time soon. (I'm really valuing my work-life balance.)

As a result, I suppose I'm open to ideas. This bug is immediately practical.

However, it would be nicer to have the plugins use a unified Koha scheduler rather than one special for plugins. 

To that end, Martin's idea for a database table that tracks tasks and execution time, and then having a scheduler (either cron-based or a separate daemon) to poll that database table for tasks every 1 minute is probably reasonable. I'm not going to work on it, but I can support that idea. 

It's a fine line between practical and technical debt but... I think we need some kind of scheduling.
Comment 58 Tomás Cohen Arazi (tcohen) 2020-12-15 15:54:49 UTC
We shouldn't make this tiny but useful enhancement depend on something as big as having the task queue code adapted to handle scheduled tasks, defined by plugins (the background jobs stuff isn't prepared yet to be extended by plugins).

It is like requiring we rewrite C4::Search to allow plugins to perform searches. While it would be awesome, it is way too much.
Comment 59 Kyle M Hall (khall) 2020-12-15 17:50:06 UTC
(In reply to Tomás Cohen Arazi from comment #58)
> We shouldn't make this tiny but useful enhancement depend on something as
> big as having the task queue code adapted to handle scheduled tasks, defined
> by plugins (the background jobs stuff isn't prepared yet to be extended by
> plugins).

Agreed, especially considering we can adapt it later.
Comment 60 David Cook 2020-12-15 22:42:03 UTC
(In reply to Tomás Cohen Arazi from comment #58)
> It is like requiring we rewrite C4::Search to allow plugins to perform
> searches. While it would be awesome, it is way too much.

I like the way that you phrased that. 

While it's great that we have the task queue now, I think that we still have a lot of work to do and it hasn't really been battle tested yet either.

Should this go back to "Signed Off" or "Passed QA"?
Comment 61 Fridolin Somers 2021-01-08 06:54:21 UTC
Please add to https://wiki.koha-community.org/wiki/Koha_Plugin_Hooks
Comment 62 Martin Renvoize (ashimema) 2021-01-11 11:28:52 UTC
Done, thanks for the reminder :)
Comment 63 Jonathan Druart 2021-01-12 15:17:08 UTC
Cannot push, release notes needed.
Comment 64 Jonathan Druart 2021-01-20 12:51:06 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 65 Fridolin Somers 2021-01-25 15:21:00 UTC
Should this be backported in 20.11.x ?
Its quite standalone so risk is minimum right ?
Comment 66 Martin Renvoize (ashimema) 2021-01-27 08:43:07 UTC
Hmm, I think it's probably simple enough for people to self-port if they need it.
Comment 67 Katrin Fischer 2021-01-27 08:46:51 UTC
I'd really like to see it backported. Kinda the point of the plugin system is not to make it necessary to change core code :) And 20.11 is still pretty young, I think it would be a nice addition and there is some history for backporting plugin and REST API things.
Comment 68 Fridolin Somers 2021-01-29 13:06:02 UTC
Patches apply so I backport.
I'm also a fan of plugins, having the cron is a must-have ;)

Note that I made a PR for a script in InLibro Carrousel plugin : 
https://github.com/inLibro/inlibro-koha/pull/8
It can now be replaced with this hook :D

Pushed to 20.11.x for 20.11.03
Comment 69 Fridolin Somers 2021-01-29 13:07:26 UTC
I updated https://wiki.koha-community.org/wiki/Koha_Plugin_Hooks
Comment 70 Andrew Fuerste-Henry 2021-02-03 22:03:58 UTC
New feature, so not backporting to 20.05. Please request if needed.
Comment 71 David Cook 2021-02-09 22:21:43 UTC
(In reply to Fridolin Somers from comment #68)
> Pushed to 20.11.x for 20.11.03

Woo!
Comment 72 Jonathan Druart 2021-03-11 11:08:41 UTC
This will be reverted if bug 27750 is not pushed ASAP.
Comment 73 Martin Renvoize (ashimema) 2021-03-11 11:16:35 UTC
Did you mean bug 27820
Comment 74 Jonathan Druart 2021-03-11 13:09:28 UTC
(In reply to Martin Renvoize from comment #73)
> Did you mean bug 27820

Yes! Thx
Comment 75 Fridolin Somers 2021-03-12 13:06:11 UTC
(In reply to Jonathan Druart from comment #74)
> (In reply to Martin Renvoize from comment #73)
> > Did you mean bug 27820
> 
> Yes! Thx

bug 27820 is pushed to master and 20.11.x
ouf ;)