Bug 32572 - We should have a background queue for each job type
Summary: We should have a background queue for each job type
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low major with 20 votes (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on: 32594
Blocks: 35092
  Show dependency treegraph
 
Reported: 2023-01-05 15:33 UTC by Nick Clemens
Modified: 2023-10-18 14:40 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2023-01-05 15:33:03 UTC
In production we are seeing that ES updates and real time holds queue can overwhelm the background worker and fall behind from "real time"

If we had a dedicated worker for each of these jobs we could batch the operations, e.g.

Fetch ES jobs until we hit 100 records or run out of queue, then index 100 at a time
Indexing 1 record at a time takes a long time for each record
Comment 1 Jonathan Druart 2023-01-06 14:00:31 UTC
(In reply to Nick Clemens from comment #0)
> Fetch ES jobs until we hit 100 records or run out of queue, then index 100
> at a time
> Indexing 1 record at a time takes a long time for each record

Isn't the problem koha-side actually, where we should batch update the index instead of call index_records for each record?
Comment 2 Nick Clemens 2023-01-06 14:13:52 UTC
(In reply to Jonathan Druart from comment #1)
> (In reply to Nick Clemens from comment #0)
> > Fetch ES jobs until we hit 100 records or run out of queue, then index 100
> > at a time
> > Indexing 1 record at a time takes a long time for each record
> 
> Isn't the problem koha-side actually, where we should batch update the index
> instead of call index_records for each record?

The problem is not for a batch action, but for a series of individual actions i.e. 10 checkins happening one after another, then scale that to a site with 50 branches, all checking in their dropbox - 500 single index actions enqueued
Comment 3 Emily Lamancusa 2023-02-10 15:21:04 UTC
I'm upping the severity of this bug as well as bug 32558, because being limited to 2 background worker processes is causing us some pretty significant issues with the volume of background jobs we generate - we've had to turn the real time holds queue off because the workers couldn't keep up.
Comment 4 Laura Escamilla 2023-09-06 16:43:48 UTC
*** Bug 34596 has been marked as a duplicate of this bug. ***
Comment 5 David Cook 2023-09-07 00:19:20 UTC
So bug 32594 introduced an "elastic_index" queue and a worker that could batch together indexing jobs.

Bug 32558 added forking so that workers could process more jobs. 

The idea here is that we'd have a "real_time_holds_queue" so that a worker could just be devoted to that instead of competing with other jobs?

--

We do need to work on a better way of setting up workers, because a smaller library will have more limited computing power, and probably will need fewer workers to handle all the jobs, while larger libraries with more computing power will want to have more workers for more separate queues...