Once Perl allocates memory, it doesn't return that memory to the Operating System for the life of that Perl process. If memory is freed within a Perl program (nullifying a variable, for instance), that memory will be re-used within the Perl program, but it will not return the memory. This can cause unexpected excessive memory consumption for long-running Perl programs. misc/background_jobs_worker.pl runs all its code within the same process, so it will never return allocated memory. This means that 1 large background import could cause background_jobs_worker.pl to permanently use a large amount of memory (until it is restarted). The solution is to do the background work within a forked process. I was reminded of this when I read "The worker process will fork a new process for every job that is being processed. This allows for resources such as memory to be returned to the operating system once a job is finished. Perl fork is very fast, so don't worry about the overhead" at https://docs.mojolicious.org/Minion. It's the same premise as bug 27267.
Some food for thought at https://stackoverflow.com/questions/9733146/tips-for-keeping-perl-memory-usage-low/9734699#9734699
Haha, I was thinking along the same lines as you! As an aside, I think forking would also resolve bug 27782 right?
(In reply to Martin Renvoize from comment #2) > Haha, I was thinking along the same lines as you! There's a reason we're friends. Haha. > As an aside, I think forking would also resolve bug 27782 right? Yeah, it would resolve bug 27782 too. Double win.
Created attachment 117500 [details] [review] Bug 27756: Fork background jobs to prevent memory leak
Created attachment 119625 [details] [review] Bug 27756: Fork background jobs to prevent memory leak Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Works as expected to me.. but I'm not an expert in forking so look to someone more familiar with that for a QA. Signing off
Created attachment 120661 [details] [review] Bug 27756: Fork background jobs to prevent memory leak Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Pushed to master for 21.05, thanks to everybody involved!
Do you think this should be pushed to 20.11.x ?
*** Bug 27782 has been marked as a duplicate of this bug. ***