Bug 31561 - Background jobs stuck at "New"
Summary: Background jobs stuck at "New"
Status: RESOLVED DUPLICATE of bug 31245
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: 21.11
Hardware: All All
: P3 normal (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-09-14 11:18 UTC by Magnus Enger
Modified: 2022-12-02 14:10 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
[Do not push] Requeue bg jobs in not found in db (2.19 KB, patch)
2022-10-28 07:42 UTC, Björn Nylén
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Magnus Enger 2022-09-14 11:18:53 UTC
"Bug 30172 - Background jobs failing due to race condition" has been fixed in 22.05.00,21.11.05,21.05.14,20.11.18, but I am still seeing background jobs stuck at "New" on 21.11.11, and there is no way to make them run apart from this: 

$ sudo koha-worker --stop <sitename>
$ sudo service rabbitmq-server stop
$ sudo koha-worker --start <sitename>

$ sudo koha-worker --stop <sitename>
$ sudo service rabbitmq-server start
$ sudo koha-worker --start <sitename>

Library staff are finding this annoying.
Comment 1 Magnus Enger 2022-09-14 11:20:45 UTC
I'd say there are two problems here: 

1. That the background jobs get stuck at "New"
2. That they are not "retried" automatically
Comment 2 Katrin Fischer 2022-09-14 11:22:35 UTC
Could this be the diacritics issue? Bug 31351
Comment 3 Aleisha Amohia 2022-09-29 00:29:06 UTC
Just upped the importance of this slightly - we have background jobs getting stuck because of Bug 31223, and then sitting at 'New'. Sometimes restarting all the services isn't the solution so would like to see a way for the jobs to get 're-picked'.
Comment 4 Björn Nylén 2022-09-30 08:00:14 UTC
We're also seeing jobs stuck at NEW. From the log it seems the worker is getting the job from the queue before it is avaliable in the db. (We're on an external db-cluster wich may compound it) 

The job will be "ack"ed in the queue even if the process call (->process on undef) fails in the worker child.

We did a hack that checks if the job exists in the db and requeues the job in the queue when not. Risk of ininite loop if it never shows up though. Not sure how big though, not live yet.

Doesn't sound like Magnus' issue thou as I don't think a restart of rabbitmq would solve this as the job is already "ack"ed there
Comment 5 Aleisha Amohia 2022-10-12 20:13:43 UTC
This is still an issue even with the release of 22.05.05.
Comment 6 Björn Nylén 2022-10-28 07:42:34 UTC
Created attachment 142746 [details] [review]
[Do not push] Requeue bg jobs in not found in db

Re-reading Magnus' description it seems to be the same issue we have. Running the bg-worker without RabbitMQ running woll fetch any new jobs from the db.

Adding hoow we work around the issue by requeuing the job. Wen testing we saw up 15+ requeues (again external db-cluster). Now after beeing live a week I don't have any stray 'new' jobs in the db.
Comment 7 Andreas Jonsson 2022-11-09 08:38:22 UTC
The issue here is that when queuing a background job while a transaction is active the transaction might not have been committed when the worker process fetches the job object from the database.

For example C4::AddBiblio, where a background job for indexing ElasticSearch is initiated from within a call to txn_do.
Comment 8 jesus 2022-11-15 22:00:14 UTC
Here are my two cents:
After restarting the server, jobs are still stuck. This is the log after restarting with the jobs stuck at New:

[Tue Nov 15 22:40:07.697358 2022] [cgi:error] [pid 2005] [client 194.126.225.130:53851] AH01215: Can't use an undefined value as an ARRAY reference at /usr/share/koha/lib/Koha/BackgroundJob/BatchUpdateItem.pm line 175.: /usr/share/koha/intranet/cgi-bin/admin/background_jobs.pl, referer: http://rgpi-intra.koha.sibadoc.es/cgi-bin/koha/admin/background_jobs.pl
[Tue Nov 15 22:40:07.731067 2022] [cgi:error] [pid 2005] [client 194.126.225.130:53851] End of script output before headers: background_jobs.pl, referer: http://rgpi-intra.koha.sibadoc.es/cgi-bin/koha/admin/background_jobs.pl

This is the error when cancelling one stuck job:

[Tue Nov 15 22:50:02.192227 2022] [cgi:error] [pid 2622] [client 194.126.225.130:53880] AH01215: Can't use an undefined value as an ARRAY reference at /usr/share/koha/lib/Koha/BackgroundJob/BatchUpdateItem.pm line 175.: /usr/share/koha/intranet/cgi-bin/admin/background_jobs.pl, referer: http://rgpi-intra.koha.sibadoc.es/cgi-bin/koha/admin/background_jobs.pl?op=cancel&id=1
[Tue Nov 15 22:50:02.228410 2022] [cgi:error] [pid 2622] [client 194.126.225.130:53880] End of script output before headers: background_jobs.pl, referer: http://rgpi-intra.koha.sibadoc.es/cgi-bin/koha/admin/background_jobs.pl?op=cancel&id=1
Comment 9 Björn Nylén 2022-11-16 06:36:42 UTC
(In reply to jesus from comment #8)
> Here are my two cents:
> After restarting the server, jobs are still stuck. This is the log after
> restarting with the jobs stuck at New:
> 
These errors seem to be related to Bug 31245. It's related to reporting on non started job but the error is the same.
Comment 10 Jonathan Druart 2022-12-02 14:10:04 UTC

*** This bug has been marked as a duplicate of bug 31245 ***