Bug 32572

Summary: We should have a background queue for each job type
Product: Koha Reporter: Nick Clemens (kidclamp) <nick>
Component: Architecture, internals, and plumbingAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: bowens, dcook, emily.lamancusa, jonathan.druart, Laura.escamilla, madamyk, phil
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32558
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 32594    
Bug Blocks: 35092    

Description Nick Clemens (kidclamp) 2023-01-05 15:33:03 UTC
In production we are seeing that ES updates and real time holds queue can overwhelm the background worker and fall behind from "real time"

If we had a dedicated worker for each of these jobs we could batch the operations, e.g.

Fetch ES jobs until we hit 100 records or run out of queue, then index 100 at a time
Indexing 1 record at a time takes a long time for each record
Comment 1 Jonathan Druart 2023-01-06 14:00:31 UTC
(In reply to Nick Clemens from comment #0)
> Fetch ES jobs until we hit 100 records or run out of queue, then index 100
> at a time
> Indexing 1 record at a time takes a long time for each record

Isn't the problem koha-side actually, where we should batch update the index instead of call index_records for each record?
Comment 2 Nick Clemens (kidclamp) 2023-01-06 14:13:52 UTC
(In reply to Jonathan Druart from comment #1)
> (In reply to Nick Clemens from comment #0)
> > Fetch ES jobs until we hit 100 records or run out of queue, then index 100
> > at a time
> > Indexing 1 record at a time takes a long time for each record
> 
> Isn't the problem koha-side actually, where we should batch update the index
> instead of call index_records for each record?

The problem is not for a batch action, but for a series of individual actions i.e. 10 checkins happening one after another, then scale that to a site with 50 branches, all checking in their dropbox - 500 single index actions enqueued
Comment 3 Emily Lamancusa 2023-02-10 15:21:04 UTC
I'm upping the severity of this bug as well as bug 32558, because being limited to 2 background worker processes is causing us some pretty significant issues with the volume of background jobs we generate - we've had to turn the real time holds queue off because the workers couldn't keep up.
Comment 4 Laura Escamilla 2023-09-06 16:43:48 UTC
*** Bug 34596 has been marked as a duplicate of this bug. ***
Comment 5 David Cook 2023-09-07 00:19:20 UTC
So bug 32594 introduced an "elastic_index" queue and a worker that could batch together indexing jobs.

Bug 32558 added forking so that workers could process more jobs. 

The idea here is that we'd have a "real_time_holds_queue" so that a worker could just be devoted to that instead of competing with other jobs?

--

We do need to work on a better way of setting up workers, because a smaller library will have more limited computing power, and probably will need fewer workers to handle all the jobs, while larger libraries with more computing power will want to have more workers for more separate queues...