Summary: | Use multi-core compression to speed up Koha backups | ||
---|---|---|---|
Product: | Koha | Reporter: | Mason James <mtj> |
Component: | Command-line Utilities | Assignee: | Mason James <mtj> |
Status: | In Discussion --- | QA Contact: | Testopia <testopia> |
Severity: | enhancement | ||
Priority: | P5 - low | CC: | jonathan.druart, m.de.rooy, mtompset, robin, tomascohen |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | --- | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: | Circulation function: | ||
Attachments: |
Bug 20543: Use multi-core compression to speed up Koha backups
Bug 20543: Use multi-core compression to speed up Koha backups |
Description
Mason James
2018-04-09 06:36:30 UTC
Created attachment 73848 [details] [review] Bug 20543: Use multi-core compression to speed up Koha backups to test... 1/ run backup, observe execution time $ sudo time koha-dump mylib real 0m17.000s 2/ apply patch 3/ copy new koha-dump to sbin $ sudo cp debian/scripts/koha-dump /usr/sbin/koha-dump 4/ install pigz $ sudo apt-get install pigz 5/ run backup, observer execution time is less $ sudo time koha-dump mylib real 0m8.000s * tested on a 4 thread system Nice! I used perl /home/vagrant/misc4dev/cp_debian_files.pl to place the script in place. And toggled between master and my patched branch. while true; do time sudo koha-dump kohadev; done -- had to CTRL-Z and then kill %1 to stop it, but the idea was to get a feel for multiple runs, since I was just using the default kohadev DB. Basically 1.3 (old) vs 0.8 (patched) real seconds on a laptop with 4 cores 2 threads, virtualizing the jessie box with "4" of the "8" cpus (tweaked the virtualbox settings before testing this). Times were comparable when the box only had 1 cpu. Created attachment 73927 [details] [review] Bug 20543: Use multi-core compression to speed up Koha backups to test... 1/ run backup, observe execution time $ sudo time koha-dump mylib real 0m17.000s 2/ apply patch 3/ copy new koha-dump to sbin $ sudo cp debian/scripts/koha-dump /usr/sbin/koha-dump 4/ install pigz $ sudo apt-get install pigz 5/ run backup, observer execution time is less $ sudo time koha-dump mylib real 0m8.000s * tested on a 4 thread system Signed-off-by: Mark Tompsett <mtompset@hotmail.com> Speed up.. Sounds like an enhancement to me. Correct me when I am wrong One more deps, one more deps! :) Maybe we should use it only if installed (and not require it?) (In reply to Jonathan Druart from comment #5) > One more deps, one more deps! :) > > Maybe we should use it only if installed (and not require it?) It's not installed by default. I think adding the dependency is a good idea, as we want faster out of the box, not requiring knowing the magical incantation to get it to work. (In reply to M. Tompsett from comment #6) > It's not installed by default. Hum? The patch adds the dependency. In debian/control, I read "koha-common depends on pigz" (In reply to Jonathan Druart from comment #7) > (In reply to M. Tompsett from comment #6) > > It's not installed by default. > > Hum? The patch adds the dependency. In debian/control, I read "koha-common > depends on pigz" Right, and it should add it, because it is kind of silly to have a patch that decides which to use if pigz is installed. People would generally not read any documentation, if there would be any, about that possibility. Because we want this out of the box, not need to know arcane knowledge to use it. This patch is just fine the way it is. (In reply to Jonathan Druart from comment #7) > (In reply to M. Tompsett from comment #6) > > It's not installed by default. > > Hum? If we didn't add the dependency, it isn't in debian by default. You have to install it. (In reply to Jonathan Druart from comment #5) > One more deps, one more deps! :) > > Maybe we should use it only if installed (and not require it?) hi Jonathan a single 56k dependency, for a 3x speedup in backup time (i7-2600 cpu) why would anyone *not* want this enabled, by default? Reducing the number of dependencies is on my long-term todo list, I am keeping it in mind. We could add that tips to the manpage. I am not strongly attach to it, I just wanted to get opinions on the idea. If you all think it was a bad idea, let's add a new dependency! :) Is pigz available on D8, D9 and supported Ubuntus? (I guess 16.04 and 18.04). While I see the advantage of having backups done faster, they are also an async task, and using all available cores in full (as a peak) might not be good for production sites. No consensus here yet - moving to 'in Discussion'. |