| Summary: | DB backup improvements: use ZSTD compression and tweak default settings | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Michał <schodkowy.omegi-0r> |
| Component: | Architecture, internals, and plumbing | Assignee: | Michał <schodkowy.omegi-0r> |
| Status: | Needs Signoff --- | QA Contact: | Testopia <testopia> |
| Severity: | enhancement | ||
| Priority: | P5 - low | CC: | januszop, schodkowy.omegi-0r |
| Version: | Main | ||
| Hardware: | All | ||
| OS: | All | ||
| GIT URL: | Initiative type: | --- | |
| Sponsorship status: | --- | Comma delimited list of Sponsors: | |
| Crowdfunding goal: | 0 | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: | Version(s) released in: | ||
| Circulation function: | |||
| Attachments: |
[PATCH 1/3] Bug 42028: use ZTSD instead of GZIP for koha-dump
[PATCH 2/3] Bug 42028: tweak default koha-run-backups settings [PATCH 3/3] Bug 42028: set process niceness for koha-dump |
||
|
Description
Michał
2026-03-09 14:29:32 UTC
Created attachment 195003 [details] [review] [PATCH 1/3] Bug 42028: use ZTSD instead of GZIP for koha-dump [PATCH 1/3] Bug 42028: use ZTSD instead of GZIP for koha-dump Notes: 1. By default ZSTD is much faster, while also providing smaller files 2. In koha-run-backups I've reorganized the deletion to be more readable and support both old and new extension. The instance name is properly escaped juuust in case, and a single regex is used. Mtime option was simplified, it works the same way while being more lean. These shell expansions were tested in both bash and dash, so /bin/sh shebang should be fine to remain. Even though --days must be greater than 0, if someone removed that check, `-mtime +-1` would still work as expected, amazingly. --- Benchmarking: # time koha-dump biblioteka Dumping Koha site biblioteka: * DB to /var/spool/koha/biblioteka/biblioteka-2026-03-09.sql.gz * configs, logs to /var/spool/koha/biblioteka/biblioteka-2026-03-09.tar.gz Done. real 1m13.647s user 1m22.566s sys 0m4.807s # time koha-dump biblioteka Dumping Koha site biblioteka: * DB to /var/spool/koha/biblioteka/biblioteka-2026-03-09.sql.zst * configs, logs to /var/spool/koha/biblioteka/biblioteka-2026-03-09.tar.zst Done. real 0m15.590s user 0m22.709s sys 0m5.305s # ls -lh /var/spool/koha/biblioteka/ | grep -F '2026-03-09.' -rw-r----- 1 root biblioteka-koha 111M Mar 9 15:20 biblioteka-2026-03-09.sql.gz -rw-r----- 1 root biblioteka-koha 99M Mar 9 15:23 biblioteka-2026-03-09.sql.zst -rw-r----- 1 root biblioteka-koha 949M Mar 9 15:21 biblioteka-2026-03-09.tar.gz -rw-r----- 1 root biblioteka-koha 937M Mar 9 15:23 biblioteka-2026-03-09.tar.zst Created attachment 195004 [details] [review] [PATCH 2/3] Bug 42028: tweak default koha-run-backups settings [PATCH 2/3] Bug 42028: tweak default koha-run-backups settings "Configuration" tar file should not be so outrageously big (~10x bigger than the database, what in the world!). The biggest offender is the Zebra index files. These should not be backed up at all, instead regenerate them after restoring backup. There's no consistency between SQL and index dumps, they're not atomic, so restoring them like that is unsafe anyway. In fact, ideally that option should be removed completely so that nobody can shoot their foot off. Logs these days can also grow quite big with all these crazy AI crawlers out there nowadays and default retention settings. I don't think making N copies of the very same logs on the server, which are meant to be ultimately ephemeral anyways, is really beneficial... And now that we shrunk the needed storage by ~10x (YMMV), bump default 2 days to 7, making it much more likely you'll have old enough point of reference in case anything goes wrong with your Koha, before it gets nuked forever. Created attachment 195005 [details] [review] [PATCH 3/3] Bug 42028: set process niceness for koha-dump [PATCH 3/3] Bug 42028: set process niceness for koha-dump Check niceness with htop, check ionice with: ionice -p `pidof zstd` (or replace zstd with mysqldump or tar) Test plan: PATCH 1: 1. Before applying run `time koha-dump INSTANCE` 2. Apply patch 3. Run #1 again and compare the times, run `ls -lh /var/spool/koha/INSTANCE` and compare the file sizes for .gz and .zst Optionally you may run `koha-run-backups --days 7 --exclude-indexes --exclude-logs --output /var/spool/koha` to make sure it works as expected, altering file name and date with `touch -d` or so, to make sure the --days option still works as expected and old files are getting removed. PATCH 2: 1. Run old cron command: `koha-run-backups --days 2 --output /var/spool/koha` 2. Look at the generated file size of .tar.zst in /var/spool/koha/INSTANCE 3. Run new command: `koha-run-backups --days 7 --exclude-indexes --exclude-logs --output /var/spool/koha` 4. Compare the sizes. Note that production installations with sizable Zebra index and log files will only yield the proper difference effect. PATCH 3: Run `koha-dump INSTANCE` and in another terminal use a tool like htop and you can sort by "NI" column to make sure koha-dump and its children processes have value of 10. You may also check ionice with: ionice -p `pidof zstd` (or replace zstd with mysqldump or tar). Be quick, since zstd backup finished much more quickly, you'd need to check these values before it finishes. The error messages for renice and ionice should not be shown at the top of your koha-dump terminal output (own process normally can adjust own values, and also these scripts are in sbin to only run as root...). |