Bug 17690

Summary:	Use task scheduler daemon
Product:	Koha	Reporter:	David Cook <dcook>
Component:	Task Scheduler	Assignee:	David Cook <dcook>
Status:	CLOSED WONTFIX	QA Contact:	Testopia <testopia>
Severity:	enhancement
Priority:	P5 - low	CC:	andreas.hedstrom.mace, josef.moravec, ztajoli
Version:	Main
Hardware:	All
OS:	All
See Also:	https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=10662
Change sponsored?:	---	Patch complexity:	---
Documentation contact:		Documentation submission:
Text to go in the release notes:	This a task scheduler daemon, which will schedule and execute tasks in the background. In terms of scheduling, it uses specific timestamps, and you can set up periodic tasks by defining the number of seconds to wait until you execute the task again. Now a cron-style syntax is not present.	Version(s) released in:
Circulation function:

Description David Cook 2016-11-29 04:33:59 UTC

As part of my work for #10662, I've been building a task scheduler daemon, which will schedule and execute tasks in the background.

Originally, I was going to include it in that patch, but it's a hefty daemon, so I've decided to open up a separate ticket.

There are a few existing bugs that talk about task schedulers (#1993, #14712, #13813, #3935), but nothing has really been proposed yet.

So I wrote my own task scheduler daemon, which I've dubbed Icarus. The master process is event-driven using POE with a single thread, and tasks are executed in child processes using POE::Wheel::Run. You can communicate with Icarus by sending JSON requests over a Unix socket.

At the moment, I've only written tasks to download and upload OAI-PMH documents to a API (defined in #10662), but you could use it for anything.

In terms of scheduling, it uses specific timestamps, and you can set up periodic tasks by defining the number of seconds to wait until you execute the task again. In the future, I'd like to add a cron-style syntax, which allows you to have a more granular control over the periodicity.

It's similar in concept to Celery (http://www.celeryproject.org/), although Celery is written in Python and much more mature. I haven't found a Perl-based project which is as full-featured. While FIFO job queues abound, a periodic task scheduler seems quite a bit more rare.

I'm going to post the API to #10662 soon, and after that I'll look at posting Icarus here. I still have more unit tests to write for it in the meantime.

Comment 1 David Cook 2016-11-29 04:40:36 UTC

At the moment, I've coded it within the Koha:: namespace, but I've been thinking about putting it into the Icarus:: or Local::Icarus:: namespace. 

I've also been thinking about converting it into a Koha plugin, so that people can try it out without fully committing to the code.

I designed Icarus to be separate from Koha. The idea is that you send a message to Icarus, and it runs a task with arguments from the message. This works well for OAI-PMH as I download records from remote servers and then send them to a Koha HTTP API, although it might not work so well with things like scheduling reports to be run and emailed. Unless you're using the report API.

Comment 2 David Cook 2017-02-21 03:41:54 UTC

I haven't received any comments about task scheduling here or on the listserv. 

Long ago, Chris Cormack mentioned Gearman, but I had my reservations. The documentation was very spotty, the project seemed dormant (although it appears to have been revived in mid 2016), and it didn't offer time-based scheduling or the level of control that we were wanting.

Martin Renvoize said there were other modules out there, but all the job servers and such I've seen don't really fit what we want.

Celery has great documentation and has a time-based scheduler, but it requires the server and workers to be written in Python, which isn't great for Koha since it's a Perl-based app.

I have a couple different Perl-based implementations I've written using POE, which work similarly to Celery, but the server and workers are written in Perl. It's not as comprehensive as Celery, but it works and it's exactly to purpose.

I'm hoping that people will accept the Perl-based design. I'm in the process of breaking it down into multiple modules which are easier to unit test. 

I'd like to get this into Koha before bug 10662, so that this can be the basis for the OAI-PMH harvesting. Maybe I'll make a simple email worker as a proof of concept and a way of evaluating the modules separate from the OAI-PMH harvesting of bug 10662...

Comment 3 David Cook 2017-07-13 04:30:59 UTC

I'm abandoning this in favour of just doing everything in a to-purpose program.