Bug 36721 - Preload core libraries to speed up worker startup and reduce memory usage
Summary: Preload core libraries to speed up worker startup and reduce memory usage
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 28410
  Show dependency treegraph
 
Reported: 2024-04-30 01:50 UTC by David Cook
Modified: 2025-03-10 12:15 UTC (History)
2 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 36721: Add -M Koha::Preload to preload libraries into starman master (2.55 KB, patch)
2024-04-30 06:31 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2024-04-30 01:50:38 UTC
If we preload core libraries using Starman's -M option, we can in theory speed up worker startup and reduce memory usage.
Comment 1 David Cook 2024-04-30 01:57:17 UTC
I've done some experimenting by using a "-M Koha::Preload" in /usr/sbin/koha-plack

You can observe memory savings using "free -m" with the most obvious memory savings being over a large number of workers. 

Speed... if just preloading Plack and Mojolicious libraries, the speed difference is negligible. 

The biggest savings in time/memory I've noticed so far is preloading Koha::Schema which preloads all our 272 DBIx::Class schema files. 

--

If we want to preload other Koha files, I think we'll need to be careful to make sure we're not causing any variable scoping problems.
Comment 2 David Cook 2024-04-30 02:41:48 UTC Comment hidden (obsolete)
Comment 3 David Cook 2024-04-30 06:31:32 UTC
Created attachment 165826 [details] [review]
Bug 36721: Add -M Koha::Preload to preload libraries into starman master

By preloading common libraries into starman master, we save memory
as starman master will share the memory addresses with its starman worker
processes.

We will also save time because each individual worker won't need to load
these common modules. They'll already be done!

To test:
0. Don't apply the patch yet
1. In koha-conf.xml, set the plack workers to a high number
2. Restart Plack
3. Note the memory usage using "free -m"

4. Apply the patch
5. Restart Plack using "debian/scripts/koha-plack"
6. Note the memory usage using "free -m"
7. Memory usage should be significantly less (around 1GB with 16 workers)

8. Comparing time is more difficult, but can be done before/after patch
using strace, top, or by modifying Koha/REST/V1.pm to output a timestamp
at the end of its load. Other ideas are also welcome.
Comment 4 David Cook 2024-04-30 06:33:36 UTC
I don't have benchmarks available at the moment, especially as I was trying out a variety of things.

But with 16 workers I was able to load Koha about 2x faster when using "-M Koha::Preload".

I think resolving bug 36722 will be another big time/speed win...
Comment 5 David Cook 2024-04-30 07:07:43 UTC
(Note: I preload an entire Catalyst app in mod_perl, and it makes for lightning fast worker startup, as all the slow stuff is done in the master process before forking.)
Comment 6 David Cook 2024-05-22 06:46:47 UTC
Note that in https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/497#note_1915557285 Victor and I talk a bit about using --preload-app with Starman to preload the entire app.

The warnings against doing this on Starman documentation make me reluctant to do that, but we could investigate that too.

Plackup seems to preload the whole app, so if we use that in dev and don't have any issues... it would be tempting to start preloading the app in prod as well, because it would likely have good performance benefits.
Comment 7 David Cook 2024-08-26 02:01:50 UTC
I'm thinking of trying this one out after Kohacon. 

I've found huge performance improvements from bug 36546, bug37657, bug 37682, and bug 34088, but... I think we can still keep going further.
Comment 8 David Cook 2025-02-18 06:13:13 UTC
I tried this out on a server with a large number of instances but a low number of workers, and the memory savings is relatively low. 

I might do a bit more playing around to see if I can find savings. Currently, on a fresh start, a starman master is about 15MB while a starman worker is around 220MB. Surely there should be things that could be shared between the parent and child.
Comment 9 David Cook 2025-02-19 03:19:01 UTC
I had another crack at this...

I watched a "starman worker" startup using a hardcoded delay and strace, and I took note of every non-Koha Perl module that gets loaded in startup, and there are about 90. 

__Memory savings__
After adding the 90 modules to Koha::Preload, I did manage to get a significant drop in RAM used across a large number of instances (about 4-5GB), even when each instance has a small number of workers.

__Startup time__
The more interesting thing to note was that "koha-plack --restart --quiet $(koha-list --enabled --plack))" took 4 minutes instead of 1 minute. 

So preloading a lot of modules actually made the overall restart slower! I think that's because koha-plack only moves on to the next instance once the starman master has launched, and it will be slower to launch since it's loading a lot of modules. 

__Startup load__
While the instances with preloaded modules were slower to load, the load was much much lower. A load of 2 where 2 workers are used. And that was very stable.

__Reload time/load__
Without preloading, you can reload a large number of instances in about 30 seconds, but it puts a huge load on the system.

With preloading, it seems the same amount of time and same load to reload. At least in terms of time reported by the "koha-plack" tool. Anecdotally, it seems like the actual reload time is faster overall, which would make sense.

--

So overall... there's pros and cons. Preloading modules means lower memory use and higher stability, but it does mean slower starts/restarts.
Comment 10 David Cook 2025-02-19 03:34:27 UTC
Technically, we wouldn't have to exclude all Koha modules. I actually do include Koha::Schema and Koha::Middleware::UserEnv as they don't have other dependencies on Koha. 

A key thing that we want to avoid is creating a socket connection in the "starman master" and re-using it in the "starman worker", so I'm trying to be extra cautious.

--

Looking at cpanfile, we have 142 modules required. Comparing it with the Koha::Preload list I made, it's fairly similar. 

--

Of course, it's also possible that preloading these modules has created subtle bugs where a module expects to just be loaded in a worker process so it sets all kinds of internal state. 

For instance, it would be interesting to look at XML::LibXSLT's register_function().

As noted on metacpan: "XML::LibXSLT has some global options. Note that these are probably not thread or even fork safe - so only set them once per process."

While I call XML::LibXSLT->register_function() in C4/XSLT.pm which isn't loaded until "starman worker" is launched, my guess would be that the actual XML::LibXSLT data structure was created in "starman master" and is shared from there. But I haven't investigated...