Bug 27756 - background_jobs_worker.pl is memory inefficient
Summary: background_jobs_worker.pl is memory inefficient
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Jonathan Druart
QA Contact: Testopia
URL:
Keywords:
: 27782 (view as bug list)
Depends on: 22417
Blocks: 28413
  Show dependency treegraph
 
Reported: 2021-02-22 23:11 UTC by David Cook
Modified: 2021-09-14 16:59 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:
21.05.00


Attachments
Bug 27756: Fork background jobs to prevent memory leak (1.64 KB, patch)
2021-03-02 10:24 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 27756: Fork background jobs to prevent memory leak (1.70 KB, patch)
2021-04-15 13:15 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 27756: Fork background jobs to prevent memory leak (1.76 KB, patch)
2021-05-07 08:29 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2021-02-22 23:11:16 UTC
Once Perl allocates memory, it doesn't return that memory to the Operating System for the life of that Perl process. If memory is freed within a Perl program (nullifying a variable, for instance), that memory will be re-used within the Perl program, but it will not return the memory. This can cause unexpected excessive memory consumption for long-running Perl programs.

misc/background_jobs_worker.pl runs all its code within the same process, so it will never return allocated memory. This means that 1 large background import could cause background_jobs_worker.pl to permanently use a large amount of memory (until it is restarted). 

The solution is to do the background work within a forked process. I was reminded of this when I read "The worker process will fork a new process for every job that is being processed. This allows for resources such as memory to be returned to the operating system once a job is finished. Perl fork is very fast, so don't worry about the overhead" at https://docs.mojolicious.org/Minion. 

It's the same premise as bug 27267.
Comment 2 Martin Renvoize 2021-02-25 09:22:09 UTC
Haha, I was thinking along the same lines as you!

As an aside, I think forking would also resolve bug 27782 right?
Comment 3 David Cook 2021-02-25 23:59:56 UTC
(In reply to Martin Renvoize from comment #2)
> Haha, I was thinking along the same lines as you!

There's a reason we're friends. Haha. 

> As an aside, I think forking would also resolve bug 27782 right?

Yeah, it would resolve bug 27782 too. Double win.
Comment 4 Jonathan Druart 2021-03-02 10:24:19 UTC
Created attachment 117500 [details] [review]
Bug 27756: Fork background jobs to prevent memory leak
Comment 5 Martin Renvoize 2021-04-15 13:15:45 UTC
Created attachment 119625 [details] [review]
Bug 27756: Fork background jobs to prevent memory leak

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 6 Martin Renvoize 2021-04-15 13:16:40 UTC
Works as expected to me.. but I'm not an expert in forking so look to someone more familiar with that for a QA.

Signing off
Comment 7 Julian Maurice 2021-05-07 08:29:36 UTC
Created attachment 120661 [details] [review]
Bug 27756: Fork background jobs to prevent memory leak

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 8 Jonathan Druart 2021-05-07 12:41:35 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 9 Fridolin Somers 2021-05-12 14:14:16 UTC
Do you think this should be pushed to 20.11.x ?
Comment 10 Jonathan Druart 2021-09-14 16:59:13 UTC
*** Bug 27782 has been marked as a duplicate of this bug. ***