If we preload core libraries using Starman's -M option, we can in theory speed up worker startup and reduce memory usage.
I've done some experimenting by using a "-M Koha::Preload" in /usr/sbin/koha-plack You can observe memory savings using "free -m" with the most obvious memory savings being over a large number of workers. Speed... if just preloading Plack and Mojolicious libraries, the speed difference is negligible. The biggest savings in time/memory I've noticed so far is preloading Koha::Schema which preloads all our 272 DBIx::Class schema files. -- If we want to preload other Koha files, I think we'll need to be careful to make sure we're not causing any variable scoping problems.
While this doesn't have to do with preloading per se, I am seeing a slowdown: I'm noticing on startup that each Starman worker spending some time doing some web requests... which are probably related to JSON::Validator. IP addresses: - 104.26.3.209 - 104.26.2.209 - 172.67.71.88 These are the IPv4 addresses for json-schema.org. I notice the first web request gets redirected as well: write(21, "GET /draft-04/schema HTTP/1.1\r\nUser-Agent: Mojolicious (Perl)\r\nAccept-Encoding: gzip\r\nHost: json-schema.org\r\n\r\n", 111) = 111 read(21, "HTTP/1.1 301 Moved Permanently\r\nDate: Tue, 30 Apr 2024 02:27:33 GMT\r\nContent-Type: text/html\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: max-age=3600\r\nExpires: Tue, 30 Apr 2024 03:27:33 GMT\r \nLocation: https://json-schema.org/draft-04/schema\r\nReport-To: {\"endpoints\":[{\"url\":\"h"..., 131072) = 870 -- Back in July 2020, there was a problem with swagger.io, which caused Koha to fail to startup. So in bug 23653 we started using a local copy of the schema. But in Bug 30194 it looks like that undid bug 23653... but that was for swagger.io. Note if json-schema.org is down (just map json-schema.org to localhost to reproduce), Koha appears to start, but the API doesn't work. You'll get 404 errors for every API endpoint. -- It looks like we can provide our own string to the $schema->specification(), so we can restore the idea behind bug 23653. That should insulate us from issues with json-schema.org and make for faster startup times.
Created attachment 165826 [details] [review] Bug 36721: Add -M Koha::Preload to preload libraries into starman master By preloading common libraries into starman master, we save memory as starman master will share the memory addresses with its starman worker processes. We will also save time because each individual worker won't need to load these common modules. They'll already be done! To test: 0. Don't apply the patch yet 1. In koha-conf.xml, set the plack workers to a high number 2. Restart Plack 3. Note the memory usage using "free -m" 4. Apply the patch 5. Restart Plack using "debian/scripts/koha-plack" 6. Note the memory usage using "free -m" 7. Memory usage should be significantly less (around 1GB with 16 workers) 8. Comparing time is more difficult, but can be done before/after patch using strace, top, or by modifying Koha/REST/V1.pm to output a timestamp at the end of its load. Other ideas are also welcome.
I don't have benchmarks available at the moment, especially as I was trying out a variety of things. But with 16 workers I was able to load Koha about 2x faster when using "-M Koha::Preload". I think resolving bug 36722 will be another big time/speed win...
(Note: I preload an entire Catalyst app in mod_perl, and it makes for lightning fast worker startup, as all the slow stuff is done in the master process before forking.)
Note that in https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/497#note_1915557285 Victor and I talk a bit about using --preload-app with Starman to preload the entire app. The warnings against doing this on Starman documentation make me reluctant to do that, but we could investigate that too. Plackup seems to preload the whole app, so if we use that in dev and don't have any issues... it would be tempting to start preloading the app in prod as well, because it would likely have good performance benefits.
I'm thinking of trying this one out after Kohacon. I've found huge performance improvements from bug 36546, bug37657, bug 37682, and bug 34088, but... I think we can still keep going further.
I tried this out on a server with a large number of instances but a low number of workers, and the memory savings is relatively low. I might do a bit more playing around to see if I can find savings. Currently, on a fresh start, a starman master is about 15MB while a starman worker is around 220MB. Surely there should be things that could be shared between the parent and child.
I had another crack at this... I watched a "starman worker" startup using a hardcoded delay and strace, and I took note of every non-Koha Perl module that gets loaded in startup, and there are about 90. __Memory savings__ After adding the 90 modules to Koha::Preload, I did manage to get a significant drop in RAM used across a large number of instances (about 4-5GB), even when each instance has a small number of workers. __Startup time__ The more interesting thing to note was that "koha-plack --restart --quiet $(koha-list --enabled --plack))" took 4 minutes instead of 1 minute. So preloading a lot of modules actually made the overall restart slower! I think that's because koha-plack only moves on to the next instance once the starman master has launched, and it will be slower to launch since it's loading a lot of modules. __Startup load__ While the instances with preloaded modules were slower to load, the load was much much lower. A load of 2 where 2 workers are used. And that was very stable. __Reload time/load__ Without preloading, you can reload a large number of instances in about 30 seconds, but it puts a huge load on the system. With preloading, it seems the same amount of time and same load to reload. At least in terms of time reported by the "koha-plack" tool. Anecdotally, it seems like the actual reload time is faster overall, which would make sense. -- So overall... there's pros and cons. Preloading modules means lower memory use and higher stability, but it does mean slower starts/restarts.
Technically, we wouldn't have to exclude all Koha modules. I actually do include Koha::Schema and Koha::Middleware::UserEnv as they don't have other dependencies on Koha. A key thing that we want to avoid is creating a socket connection in the "starman master" and re-using it in the "starman worker", so I'm trying to be extra cautious. -- Looking at cpanfile, we have 142 modules required. Comparing it with the Koha::Preload list I made, it's fairly similar. -- Of course, it's also possible that preloading these modules has created subtle bugs where a module expects to just be loaded in a worker process so it sets all kinds of internal state. For instance, it would be interesting to look at XML::LibXSLT's register_function(). As noted on metacpan: "XML::LibXSLT has some global options. Note that these are probably not thread or even fork safe - so only set them once per process." While I call XML::LibXSLT->register_function() in C4/XSLT.pm which isn't loaded until "starman worker" is launched, my guess would be that the actual XML::LibXSLT data structure was created in "starman master" and is shared from there. But I haven't investigated...