Bug 40901

Summary: koha-common.service bundles all sub-daemons under one systemd service instead of per-instance services
Product: Koha Reporter: Tristin Stagg <tristin.stagg>
Component: Architecture, internals, and plumbingAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: dcook, jonathan.druart, julian.maurice, katrin.fischer, lari.taskula, m.de.rooy, martin.renvoize, nick
Version: Main   
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:

Description Tristin Stagg 2025-09-29 18:29:06 UTC

    
Comment 1 Tristin Stagg 2025-09-29 18:33:29 UTC
Currently, the koha-common.service systemd unit is just a wrapper around /etc/init.d/koha-common, which internally uses Koha’s shell wrappers and the daemon command to launch all Koha subcomponents (Plack, Zebra, Z39.50 responder, SIP2 server, background workers, indexers, etc.) under a single systemd service.

This has several problems:

1) No process isolation: All subcomponents for all Koha instances are children of koha-common.service.

2) OOM-killer issues: When the system is under memory pressure, the Linux OOM killer can indiscriminately kill any Koha sub-daemon. In our environment, we’ve seen background workers or Plack workers being killed silently.

3) Poor observability: systemctl status koha-common shows a giant mixed tree of processes. It is difficult to identify which component is misbehaving or using excessive resources.

4) systemd integration is broken: Features like per-service cgroup limits (MemoryMax=, CPUQuota=), restart policies, logging (journalctl -u), and failure detection cannot be used at the component level.

Environment:

- Debian 10, 11, 12
- Koha version: At least 24.11-25.05

Steps to Reproduce:

1) Install Koha via Debian packages.
2) Start an instance, e.g. koha-create --create-db mainlib.
3) Run systemctl status koha-common.service.
4) Observe that all daemons (Plack, Zebra, workers, indexers, Z39.50, etc.) run under the single koha-common.service process tree.

Observed Behavior:
All Koha daemons for all instances run under one service. For example:

● koha-common.service - Start required services for each Koha instance
   Loaded: loaded (/etc/init.d/koha-common; enabled)
   Active: active (running)
   CGroup: /system.slice/koha-common.service
           ├─ starman master …
           ├─ starman worker …
           ├─ zebra …
           ├─ background_jobs_worker.pl …
           ├─ es_indexer_daemon.pl …
           └─ etc.

When memory pressure occurs, OOM-killer may kill any of these, and systemd does not restart them because they are just sub-processes of koha-common.

Expected Behavior:

Each Koha subcomponent (Plack, Zebra, workers, indexers, Z39.50, SIP2, etc.) should run as its own systemd unit, ideally with instance scoping. For example:

- koha-plack@instance.service
- koha-zebra@instance.service
- koha-z3950@instance.service
- koha-workers@instance.service
- koha-indexer@instance.service

And koha-common.service could be a meta unit (or target) that pulls in all the per-instance units.

This would allow:

- Proper systemd supervision of each daemon
- Restart policies if one crashes
- Independent resource limits
- Better logging and observability

Why this matters:
Right now, Koha administrators lose visibility and control of critical services. In production environments, a single rogue process can consume memory, trigger the OOM-killer, and kill unrelated Koha daemons, taking down parts of the system silently. This design makes Koha fragile on modern Linux systems.

Suggested Fix:

1) Replace the init.d style management with native systemd unit files for each Koha component.
2) Use templated units for instance support (koha-plack@.service, etc.).
3) Have koha-common.service or koha.target depend on these per-instance units.
4) This would align Koha with best practices for systemd-based services and improve stability, observability, and maintainability in production.
Comment 2 David Cook 2025-10-01 04:44:42 UTC
(In reply to Tristin Stagg from comment #1)

Thanks, Tristin, for all that info. I agree with a lot of what you're saying here.

That said, I think there are some challenges with it. Generally speaking, SystemD isn't available in Docker containers, so this isn't going to work for koha-testing-docker. If we were to more fully embrace SystemD, we'd need to have viable alternatives for koha-testing-docker to function. 

But if we did have alternatives for koha-testing-docker to function, it wouldn't really be testing Koha as much as we are now. I don't think that's necessarily a blocker, but it's something that would need to be carefully considered. 

> Suggested Fix:
> 
> 1) Replace the init.d style management with native systemd unit files for
> each Koha component.
> 2) Use templated units for instance support (koha-plack@.service, etc.).
> 3) Have koha-common.service or koha.target depend on these per-instance
> units.
> 4) This would align Koha with best practices for systemd-based services and
> improve stability, observability, and maintainability in production.

I have a different system I deploy using Debian packages and I have "debian/pkg.app.service" and "debian/pkg.app@.service" files where "debian/pkg.app.service" depends on "app@first.service app@second.service". It works well, but it is also built into the package itself which lives in the git repo. 

I am curious how we'd use SystemD templates with Koha's dynamic instance management with that dependency in mind. This might be possible and I'm just not familiar enough with unit files to know how to do it. It's just a concern that comes to mind at the moment. (After doing a bit of Googling, it looks like it might be possible to provide a "foo.service.wants/" directory. I suppose a person might also use an *override.conf file. I hadn't thought of that before although I've used those in other places for other reasons.)

--

So overall... I think it sounds good. The only sticking point I can think of is koha-testing-docker. But maybe that will become clearer as we re-tool the SystemD files. 

Interesting idea in any case!