Bug 20543 - Use multi-core compression to speed up Koha backups
Summary: Use multi-core compression to speed up Koha backups
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Mason James
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2018-04-09 06:36 UTC by Mason James
Modified: 2018-08-28 19:04 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 20543: Use multi-core compression to speed up Koha backups (2.41 KB, patch)
2018-04-09 08:43 UTC, Mason James
Details | Diff | Splinter Review
Bug 20543: Use multi-core compression to speed up Koha backups (2.42 KB, patch)
2018-04-10 13:09 UTC, M. Tompsett
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Mason James 2018-04-09 06:36:30 UTC
This patch uses multi-core compression to speed up Koha backups

patch coming...
Comment 1 Mason James 2018-04-09 08:43:30 UTC
Created attachment 73848 [details] [review]
Bug 20543: Use multi-core compression to speed up Koha backups

to test...

1/ run backup, observe execution time
 $ sudo time koha-dump mylib
 real    0m17.000s

2/ apply patch

3/ copy new koha-dump to sbin
 $ sudo cp debian/scripts/koha-dump /usr/sbin/koha-dump

4/ install pigz
 $ sudo apt-get install pigz

5/ run backup, observer execution time is less
 $ sudo time koha-dump mylib
 real    0m8.000s

* tested on a 4 thread system
Comment 2 M. Tompsett 2018-04-10 13:08:37 UTC
Nice! I used perl /home/vagrant/misc4dev/cp_debian_files.pl to place the script in place. And toggled between master and my patched branch.

while true; do time sudo koha-dump kohadev; done
-- had to CTRL-Z and then kill %1 to stop it, but the idea was to get a feel for multiple runs, since I was just using the default kohadev DB.

Basically 1.3 (old) vs 0.8 (patched) real seconds on a laptop with 4 cores 2 threads, virtualizing the jessie box with "4" of the "8" cpus (tweaked the virtualbox settings before testing this).

Times were comparable when the box only had 1 cpu.
Comment 3 M. Tompsett 2018-04-10 13:09:00 UTC
Created attachment 73927 [details] [review]
Bug 20543: Use multi-core compression to speed up Koha backups

to test...

1/ run backup, observe execution time
 $ sudo time koha-dump mylib
 real    0m17.000s

2/ apply patch

3/ copy new koha-dump to sbin
 $ sudo cp debian/scripts/koha-dump /usr/sbin/koha-dump

4/ install pigz
 $ sudo apt-get install pigz

5/ run backup, observer execution time is less
 $ sudo time koha-dump mylib
 real    0m8.000s

* tested on a 4 thread system

Signed-off-by: Mark Tompsett <mtompset@hotmail.com>
Comment 4 Marcel de Rooy 2018-05-11 08:03:00 UTC
Speed up.. Sounds like an enhancement to me. Correct me when I am wrong
Comment 5 Jonathan Druart 2018-05-30 20:48:11 UTC
One more deps, one more deps! :)

Maybe we should use it only if installed (and not require it?)
Comment 6 M. Tompsett 2018-05-31 02:49:55 UTC
(In reply to Jonathan Druart from comment #5)
> One more deps, one more deps! :)
> 
> Maybe we should use it only if installed (and not require it?)

It's not installed by default. I think adding the dependency is a good idea, as we want faster out of the box, not requiring knowing the magical incantation to get it to work.
Comment 7 Jonathan Druart 2018-05-31 15:30:18 UTC
(In reply to M. Tompsett from comment #6)
> It's not installed by default.

Hum? The patch adds the dependency. In debian/control, I read "koha-common depends on pigz"
Comment 8 M. Tompsett 2018-06-01 03:37:01 UTC
(In reply to Jonathan Druart from comment #7)
> (In reply to M. Tompsett from comment #6)
> > It's not installed by default.
> 
> Hum? The patch adds the dependency. In debian/control, I read "koha-common
> depends on pigz"

Right, and it should add it, because it is kind of silly to have a patch that decides which to use if pigz is installed. People would generally not read any documentation, if there would be any, about that possibility. Because we want this out of the box, not need to know arcane knowledge to use it. This patch is just fine the way it is.
Comment 9 M. Tompsett 2018-06-01 03:38:14 UTC
(In reply to Jonathan Druart from comment #7)
> (In reply to M. Tompsett from comment #6)
> > It's not installed by default.
> 
> Hum?

If we didn't add the dependency, it isn't in debian by default. You have to install it.
Comment 10 Mason James 2018-06-06 05:59:06 UTC
(In reply to Jonathan Druart from comment #5)
> One more deps, one more deps! :)
> 
> Maybe we should use it only if installed (and not require it?)

hi Jonathan
a single 56k dependency, for a 3x speedup in backup time (i7-2600 cpu)

why would anyone *not* want this enabled, by default?
Comment 11 Jonathan Druart 2018-06-06 14:30:00 UTC
Reducing the number of dependencies is on my long-term todo list, I am keeping it in mind. We could add that tips to the manpage.

I am not strongly attach to it, I just wanted to get opinions on the idea. If you all think it was a bad idea, let's add a new dependency! :)
Comment 12 Tomás Cohen Arazi 2018-08-28 18:15:39 UTC
Is pigz available on D8, D9 and supported Ubuntus? (I guess 16.04 and 18.04).

While I see the advantage of having backups done faster, they are also an async task, and using all available cores in full (as a peak) might not be good for production sites.
Comment 13 Katrin Fischer 2018-08-28 19:04:23 UTC
No consensus here yet - moving to 'in Discussion'.