If we preload core libraries using Starman's -M option, we can in theory speed up worker startup and reduce memory usage.
I've done some experimenting by using a "-M Koha::Preload" in /usr/sbin/koha-plack You can observe memory savings using "free -m" with the most obvious memory savings being over a large number of workers. Speed... if just preloading Plack and Mojolicious libraries, the speed difference is negligible. The biggest savings in time/memory I've noticed so far is preloading Koha::Schema which preloads all our 272 DBIx::Class schema files. -- If we want to preload other Koha files, I think we'll need to be careful to make sure we're not causing any variable scoping problems.
While this doesn't have to do with preloading per se, I am seeing a slowdown: I'm noticing on startup that each Starman worker spending some time doing some web requests... which are probably related to JSON::Validator. IP addresses: - 104.26.3.209 - 104.26.2.209 - 172.67.71.88 These are the IPv4 addresses for json-schema.org. I notice the first web request gets redirected as well: write(21, "GET /draft-04/schema HTTP/1.1\r\nUser-Agent: Mojolicious (Perl)\r\nAccept-Encoding: gzip\r\nHost: json-schema.org\r\n\r\n", 111) = 111 read(21, "HTTP/1.1 301 Moved Permanently\r\nDate: Tue, 30 Apr 2024 02:27:33 GMT\r\nContent-Type: text/html\r\nTransfer-Encoding: chunked\r\nConnection: keep-alive\r\nCache-Control: max-age=3600\r\nExpires: Tue, 30 Apr 2024 03:27:33 GMT\r \nLocation: https://json-schema.org/draft-04/schema\r\nReport-To: {\"endpoints\":[{\"url\":\"h"..., 131072) = 870 -- Back in July 2020, there was a problem with swagger.io, which caused Koha to fail to startup. So in bug 23653 we started using a local copy of the schema. But in Bug 30194 it looks like that undid bug 23653... but that was for swagger.io. Note if json-schema.org is down (just map json-schema.org to localhost to reproduce), Koha appears to start, but the API doesn't work. You'll get 404 errors for every API endpoint. -- It looks like we can provide our own string to the $schema->specification(), so we can restore the idea behind bug 23653. That should insulate us from issues with json-schema.org and make for faster startup times.
Created attachment 165826 [details] [review] Bug 36721: Add -M Koha::Preload to preload libraries into starman master By preloading common libraries into starman master, we save memory as starman master will share the memory addresses with its starman worker processes. We will also save time because each individual worker won't need to load these common modules. They'll already be done! To test: 0. Don't apply the patch yet 1. In koha-conf.xml, set the plack workers to a high number 2. Restart Plack 3. Note the memory usage using "free -m" 4. Apply the patch 5. Restart Plack using "debian/scripts/koha-plack" 6. Note the memory usage using "free -m" 7. Memory usage should be significantly less (around 1GB with 16 workers) 8. Comparing time is more difficult, but can be done before/after patch using strace, top, or by modifying Koha/REST/V1.pm to output a timestamp at the end of its load. Other ideas are also welcome.
I don't have benchmarks available at the moment, especially as I was trying out a variety of things. But with 16 workers I was able to load Koha about 2x faster when using "-M Koha::Preload". I think resolving bug 36722 will be another big time/speed win...
(Note: I preload an entire Catalyst app in mod_perl, and it makes for lightning fast worker startup, as all the slow stuff is done in the master process before forking.)
Note that in https://gitlab.com/koha-community/koha-testing-docker/-/merge_requests/497#note_1915557285 Victor and I talk a bit about using --preload-app with Starman to preload the entire app. The warnings against doing this on Starman documentation make me reluctant to do that, but we could investigate that too. Plackup seems to preload the whole app, so if we use that in dev and don't have any issues... it would be tempting to start preloading the app in prod as well, because it would likely have good performance benefits.
I'm thinking of trying this one out after Kohacon. I've found huge performance improvements from bug 36546, bug37657, bug 37682, and bug 34088, but... I think we can still keep going further.