Bug 35092 - [OMNIBUS] Remaining background job/worker issues
Summary: [OMNIBUS] Remaining background job/worker issues
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on: 27258 27434 30182 30654 30656 31090 31124 31336 31363 31580 31616 32243 32305 32395 32572 32736 33334 33898 34362 34477 34596 34990 34997 34998 35111 35118 35655 35819 35920
Blocks:
  Show dependency treegraph
 
Reported: 2023-10-18 14:33 UTC by Marcel de Rooy
Modified: 2024-04-04 13:11 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Marcel de Rooy 2023-10-18 14:33:09 UTC
32305
Background worker doesn't check job status when received from rabbitmq

33334
koha-worker.service need update for background job queues

34477
Further reduce memory usage for idle background job worker

34990

34997

Let me add a few more.
Comment 1 Marcel de Rooy 2023-10-19 09:04:03 UTC
Some thoughts on the sync issues between MQ and DB (bugs 32305, 34990 and 34997):

What are the different scenarios where the (Rabbit) message queue (MQ) and the background_jobs table (DB) are out of sync ? FIrst note that we can assume that enqueued jobs are always saved in the DB; a message without a job in the DB will be discarded when reading it from MQ.

[1] New jobs in DB are not found in MQ.
CAUSE: If MQ is not running, Koha only writes to the DB. If MQ is restarted, all jobs on MQ are gone too (solved by 34990).
If you start the worker after MQ in this context, the worker only processes MQ jobs. So you will have "orphaned jobs" in the DB. But be aware that they will pop up again as soon as you start the worker before MQ !
As long as you restart the worker before or without MQ, things will be "fine". Actually we are just ignoring MQ.
We need to decide how to change the algorithm in the worker to resolve this. See further below.

[2] New jobs in MQ have another status in DB
CAUSE: The worker has been started before MQ. It only processes DB jobs. So a large number of jobs is accumulating on MQ.
Without bug 34990, we can restart MQ to clear its queue and restart the worker after it processed all DB jobs to be in sync again. If we do not start MQ, we will just keep processing DB entries.
With bug 34990 and 32305, new jobs on MQ that have another status in DB will just be discarded. Restarting the worker after MQ will restore a synced situation. If we restart worker before or without MQ, the situation does not actually change.
To improve, we should not send messages to MQ if we are not processing them in the worker at this moment. See further below.

What benefit have bugs 32305 + 34990 ?
Bug 34990 reduces the number of orphaned (lost) DB jobs after a MQ restart when actually using MQ.
Bug 32305 prevents running a job that has been processed already. If the worker was in DB mode but MQ is active, a worker restart would switch to MQ mode and reprocess MQ jobs. The number could even increase when we would only have 34990 since it persists MQ jobs. The combination of 34990 and 32305 resolves that.
If you do not use MQ at all, the bugs do not change anything.
Comment 2 Marcel de Rooy 2023-10-19 09:10:22 UTC
As a result of the above, I will make 34990 (functionally) dependent of 32305. It is in PQA but should not be pushed separately.

Note that we can improve/relocate the code of 32305 but I would recommend to do that while refactoring a bit more.

Will still add a comment about options to go further with worker algorithm.
Comment 3 Marcel de Rooy 2023-10-19 09:21:53 UTC
TODO move the selection to a filter_by method in BackgroundJobs
TODO Add a MessageQueue module to move a few methods out of BackgroundJob like connect and .. ?
Comment 4 Marcel de Rooy 2023-10-19 09:22:18 UTC
(In reply to Marcel de Rooy from comment #3)
> TODO move the selection (in worker) to a filter_by method in BackgroundJobs
Comment 5 Jonathan Druart 2023-10-19 09:30:38 UTC
Did you see David Cook's findings on the ML?

https://lists.koha-community.org/pipermail/koha-devel/2023-October/048379.html
[Koha-devel] Background jobs quirks
Comment 6 Marcel de Rooy 2023-10-19 09:41:07 UTC
(In reply to Jonathan Druart from comment #5)
> Did you see David Cook's findings on the ML?
> 
> https://lists.koha-community.org/pipermail/koha-devel/2023-October/048379.
> html
> [Koha-devel] Background jobs quirks

20231018 07:44:06 instance-koha-worker-long_tasks: client (pid 3949888)
killed by signal 13, respawning
Comment 7 Marcel de Rooy 2023-10-19 10:04:24 UTC
We should decide imo how to resolve sync issues as mentioned before. I thought of a few options.

[1] Still hybrid but dynamical approach. The worker should regularly check both MQ and DB to select new jobs. So more dynamical than the current approach that is determined by MQ being active or not at the start of the worker.

[2] Stricter distinction between MQ and DB: Add a preference like UseMessageQ to indicate that the worker only reads jobs from either MQ or DB. If you choose the DB side, the MQ will no longer be filled. If you choose MQ, the worker should not be able to switch to "DB mode" anymore like it does now.

[3] Remove DB mode: This involves "trusting MQ to be stable enough". We will get new jobs from MQ, and only save processed jobs in DB.

[4] Remove MQ mode: The other way around. "We do not really trust MQ." Remove enqueuing to MQ and reading new jobs from it.

Please give some feedback. What is the best way to go forward?
Comment 8 Marcel de Rooy 2023-10-19 10:04:42 UTC
(In reply to Marcel de Rooy from comment #7)
> Please give some feedback. What is the best way to go forward?

Personally, I do not favor option 1. Code gets more complex. Problems are harder to understand/resolve. Option 2 could be an intermediary step in building enough trust to eventually go to option 3. I would favor option 3 above option 4. We chose for a message queue; I think there is no good reason to abandon it right now. We did not spend enough time on getting the issues clear and trying to address them after that.
Comment 9 Blou 2023-10-19 12:34:16 UTC
I would favor option 4, and option 2 as a lesser evil scenario.

We use the database everywhere, we keep trace of everything in it, including deleted objects, and can use it to retrace everything that happened to a library when they ask.  


And then we use an outside tool for messaging to a background worker coded by/for Koha, for tasks where a nanosecond reaction time is absolutely not necessary.



MQ (and Erlang) is surprisingly demanding on ressources for the little it offers.
The last two releases, I've hacked the background_jobs tool to make sure the database was always used, and checked at startup, so that no task was lost.  You sure can invest a lot of efforts to correct MQ, but why?  What is the long term objective for this?  Because background tasks are sure not enough to justify the added complexity.
Comment 10 David Cook 2023-10-19 23:56:53 UTC
(In reply to Marcel de Rooy from comment #7)
> We should decide imo how to resolve sync issues as mentioned before. I
> thought of a few options.
> 
> [1] Still hybrid but dynamical approach. The worker should regularly check
> both MQ and DB to select new jobs. So more dynamical than the current
> approach that is determined by MQ being active or not at the start of the
> worker.
> 
> [2] Stricter distinction between MQ and DB: Add a preference like
> UseMessageQ to indicate that the worker only reads jobs from either MQ or
> DB. If you choose the DB side, the MQ will no longer be filled. If you
> choose MQ, the worker should not be able to switch to "DB mode" anymore like
> it does now.
> 
> [3] Remove DB mode: This involves "trusting MQ to be stable enough". We will
> get new jobs from MQ, and only save processed jobs in DB.
> 
> [4] Remove MQ mode: The other way around. "We do not really trust MQ."
> Remove enqueuing to MQ and reading new jobs from it.
> 
> Please give some feedback. What is the best way to go forward?

I think option 2 is a reasonable compromise. 

Personally, my preference is for the MQ. It's a great tool I use for a variety of systems beyond Koha. I understand it well enough that I can investigate and resolve problems encountered while using it. I think the more we use it the more we'll learn and the better we'll become with it, and the further into the present/future we'll move as service providers and technologists. I think it's well known that technology moves quickly and you have to continually learn and adapt with its progress. (Another example is using Vue.js. It's an exciting client side technology that we'd all do well to learn both for the benefit of Koha and ourselves as software professionals.)

That said, I can understand that some people are more comfortable using the software tools that they already know and don't want to push boundaries or explore frontiers. There is certainly merit to sticking to battle tested tools that are well understood by a broad base. 

One of the downsides of option 2 is that it splits focus though. The Koha community has limited resources and part of our success has come from most of us implementing Koha in the same way, as we've been able to work together to advance the project. If we continue to split between MQ and DB, we might find it becomes more difficult to maintain.
Comment 11 David Cook 2023-10-19 23:57:57 UTC
I've just reported bug 35111 which I hope will improve the experience of MQ driven background jobs in Koha.

Funny enough the problem isn't with RabbitMQ per se but rather with the background jobs worker crashing because it lost its connection to the MySQL database. Food for thought there.
Comment 12 David Cook 2023-10-20 01:49:31 UTC
(In reply to David Cook from comment #11)
> I've just reported bug 35111 which I hope will improve the experience of MQ
> driven background jobs in Koha.
> 
> Funny enough the problem isn't with RabbitMQ per se but rather with the
> background jobs worker crashing because it lost its connection to the MySQL
> database. Food for thought there.

Actually, after further investigation, it looks like it was even more complicated. It looks like a difference in behaviour of MySQL C client libraries vs MariaDB C client libraries (the former used by Ubuntu and the latter used by Debian). Easy fix in the end anyway...
Comment 13 Jonathan Druart 2023-10-20 07:45:25 UTC
My proposal:
1. Push bug 35111 (because it might fix a lot of the trouble people encounters in production!) and have it part of 23.11
2. Ask people on the ML what *exact* problems they are getting with MQ and why they are not using it.
3. Write a script to reproduce the problems locally (run as root so that you can restart things):
  * Fast enqueue of thousands of request
  * Enqueue huge ES reindex
  * Deal with disconnects
  * Other problems with null/0 status
  * Other problems from the ML
4. Fix the main bugs
5. Clean up
6. Fix the easy bugs
Comment 14 Marcel de Rooy 2023-10-20 08:29:58 UTC
(In reply to Blou from comment #9)
> I would favor option 4, and option 2 as a lesser evil scenario.

Thanks for your feedback. Your point makes sense. The dilemma is that the other side makes sense too imo ;)
Comment 15 Marcel de Rooy 2023-10-20 08:31:19 UTC
(In reply to David Cook from comment #10)
> I think option 2 is a reasonable compromise. 
> 
> Personally, my preference is for the MQ. 

> One of the downsides of option 2 is that it splits focus though.

Thanks. Balanced view. There is no easy choice.
Comment 16 Marcel de Rooy 2023-10-20 08:34:05 UTC
(In reply to Jonathan Druart from comment #13)
> My proposal:
> 1. Push bug 35111 (because it might fix a lot of the trouble people
> encounters in production!) and have it part of 23.11
> 2. Ask people on the ML what *exact* problems they are getting with MQ and
> why they are not using it.
> 3. Write a script to reproduce the problems locally (run as root so that you
> can restart things):
>   * Fast enqueue of thousands of request
>   * Enqueue huge ES reindex
>   * Deal with disconnects
>   * Other problems with null/0 status
>   * Other problems from the ML
> 4. Fix the main bugs
> 5. Clean up
> 6. Fix the easy bugs

Thanks. Not sure if 35111 will address that. We can try the ML but we should have had that feedback already. Would we receive it now? "Fix the main bugs". We should first prioritize but imo we need to choose a direction there and you do not give your view about that here?
Comment 17 Jonathan Druart 2023-10-20 08:54:50 UTC
Voting for 2, then 3 (when considered stable enough).
Comment 18 Blou 2023-10-20 13:02:21 UTC
I'm all for good technologies for solving problems, but I have always had a problem with technology for technology's sake.

My personnal empowerment and my resume-building should not be reasons to rearchitecture a product such as Koha.  So what is the endgame for MQ ?  I made a genuine search for a RFC on MQ (I admit I'm bad, I could have missed it), and looked at the initial bzs, but I couldn't find what was the future use for MQ ?  All I see are the background tasks, which is a simple problem with a simple solution.  

Could someone sell it to me (and others) ?
Comment 19 Katrin Fischer 2023-12-29 09:04:26 UTC
I have filed bug 35655 asking for a proper way of switching RabbitMQ off. I feel we should have that to mitigate known issues for now, but it could also be good for some installations not to have to deal with the additional dependency and resources.

In a 23.05 I have reports especially about problems with the elasticsearch jobs. There are jobs that just remain as 'new' and never seem to get looked at. The index info shows that the index gets out of sync pretty fast after a full reindex. The system has over a million records.
Comment 20 David Cook 2024-01-24 05:31:08 UTC
(In reply to Blou from comment #18)
> Could someone sell it to me (and others) ?

The MQ works pretty much perfectly for me (at least since bug 35111), so I'd need someone to sell me not using the MQ.
Comment 21 Marcel de Rooy 2024-01-24 12:32:03 UTC
(In reply to David Cook from comment #20)
> (In reply to Blou from comment #18)
> > Could someone sell it to me (and others) ?
> 
> The MQ works pretty much perfectly for me (at least since bug 35111), so I'd
> need someone to sell me not using the MQ.

Yeah I have no problems with it either. But there is a huge cry for making the MQ optional..
The current hybrid approach has proven to be bug prone. So it may be hard to balance enhancing master and fixing stable branches..
Comment 22 Julian Maurice 2024-01-24 13:57:59 UTC
(In reply to David Cook from comment #20)
> The MQ works pretty much perfectly for me (at least since bug 35111), so I'd
> need someone to sell me not using the MQ.
If it works for you and solves a real problem, then great.
But not everyone needs it and making an external dependency like RabbitMQ mandatory should be done with care as it can add a lot of "small costs" everywhere (for devs, testers, server admins, ...).

(In reply to Marcel de Rooy from comment #21)
> The current hybrid approach has proven to be bug prone. So it may be hard to
> balance enhancing master and fixing stable branches..
By "hybrid" do you mean the fact that one script does both things (MQ and SQL polling) ? Would that be better to have two separate "worker" scripts ?
Comment 23 Marcel de Rooy 2024-01-24 14:14:40 UTC
(In reply to Julian Maurice from comment #22)
> By "hybrid" do you mean the fact that one script does both things (MQ and
> SQL polling) ? Would that be better to have two separate "worker" scripts ?

Yes and no :)
It should not need to be two scripts. But see above too. We suggest to not switch anymore between reading new jobs from DB or from MQ depending on MQ state.
In a new DB mode we do not use MQ and read only from DB. In a new MQ mode we only read new jobs from MQ, so not from DB. DB will only have job results.
An attempt to get rid of various out-of-sync problems, race conditions etc. between MQ and DB as we are having now.
Comment 24 Blou 2024-01-24 14:41:56 UTC
(In reply to Marcel de Rooy from comment #23)
> An attempt to get rid of various out-of-sync problems, race conditions etc.
> between MQ and DB as we are having now.

We fixed that by getting rid of MQ.  Flawless Victory.  :^)
Comment 25 Julian Maurice 2024-01-24 15:02:09 UTC
(In reply to Marcel de Rooy from comment #23)
> (In reply to Julian Maurice from comment #22)
> > By "hybrid" do you mean the fact that one script does both things (MQ and
> > SQL polling) ? Would that be better to have two separate "worker" scripts ?
> 
> Yes and no :)
> It should not need to be two scripts. But see above too.

Sorry, I should have read the whole thing before replying...
I think option 2 is the best for everyone (but IMO configuration should be in $KOHA_CONF, not in a syspref)
Comment 26 wajasu 2024-04-01 23:54:46 UTC
Cleaned up this old bug since it was resolved by bug 32465

*** This bug has been marked as a duplicate of bug 32465 ***
Comment 27 Jonathan Druart 2024-04-02 09:20:43 UTC
(In reply to wajasu from comment #26)
> Cleaned up this old bug since it was resolved by bug 32465
> 
> *** This bug has been marked as a duplicate of bug 32465 ***

This is an omnibus bug and should stay opened until all dependent bugs have been resolved.