Bug 29507 - Speed up auto renew cronjob via parallel processing
Summary: Speed up auto renew cronjob via parallel processing
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Nick Clemens (kidclamp)
QA Contact: Martin Renvoize
URL:
Keywords: additional_work_needed
Depends on:
Blocks:
 
Reported: 2021-11-17 17:34 UTC by Nick Clemens (kidclamp)
Modified: 2024-07-05 14:09 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.11.00


Attachments
Bug 28833: Speed up auto renew cronjob via parallel processing (15.17 KB, patch)
2021-11-17 17:35 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29057: Speed up auto renew cronjob via parallel processing (15.17 KB, patch)
2021-11-17 17:45 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (15.04 KB, patch)
2022-03-23 15:17 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (18.06 KB, patch)
2023-05-04 19:29 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (18.63 KB, patch)
2023-10-25 11:20 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.59 KB, patch)
2024-01-23 19:11 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Tidy (20.12 KB, patch)
2024-01-23 19:11 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.68 KB, patch)
2024-01-24 11:33 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 29507: Tidy (20.21 KB, patch)
2024-01-24 11:33 UTC, Matt Blenkinsop
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.66 KB, patch)
2024-04-18 13:16 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Tidy (20.24 KB, patch)
2024-04-18 13:16 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.66 KB, patch)
2024-07-02 13:00 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Tidy (20.35 KB, patch)
2024-07-02 13:00 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.72 KB, patch)
2024-07-02 15:31 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 29507: Tidy (20.41 KB, patch)
2024-07-02 15:31 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2021-11-17 17:34:35 UTC

    
Comment 1 Nick Clemens (kidclamp) 2021-11-17 17:35:06 UTC Comment hidden (obsolete)
Comment 2 Nick Clemens (kidclamp) 2021-11-17 17:45:32 UTC
Created attachment 127744 [details] [review]
Bug 29057: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values
Comment 3 Nick Clemens (kidclamp) 2022-03-23 15:17:48 UTC
Created attachment 132056 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values
Comment 4 Jonathan Druart 2023-05-04 13:19:58 UTC
Does not apply.

I am wondering how this can work with the digest option.
Comment 5 Nick Clemens (kidclamp) 2023-05-04 19:29:40 UTC
Created attachment 150692 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values
Comment 6 Nick Clemens (kidclamp) 2023-05-04 19:34:59 UTC
(In reply to Jonathan Druart from comment #4)

> I am wondering how this can work with the digest option.

There's a loop to split based on borrowers. We push all the issues from the first borrower to chunk 0, when we complete one borrower we push the next on to the next chunk, then the next borrower to next chunk

When we have pushed to as many chunks as we need (as many loops) we start again.

So a single borrower, no matter how many checkouts, will always be processed by a single fork. It means the chunks won't be necessarily the same size, but they will work with digest
Comment 7 Nick Clemens (kidclamp) 2023-10-25 11:20:46 UTC
Created attachment 157772 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values
Comment 8 Nick Clemens (kidclamp) 2024-01-23 19:11:16 UTC
Created attachment 161314 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values
Comment 9 Nick Clemens (kidclamp) 2024-01-23 19:11:18 UTC
Created attachment 161315 [details] [review]
Bug 29507: Tidy
Comment 10 Matt Blenkinsop 2024-01-24 11:33:52 UTC
Created attachment 161326 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 11 Matt Blenkinsop 2024-01-24 11:33:55 UTC
Created attachment 161327 [details] [review]
Bug 29507: Tidy

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 12 Marcel de Rooy 2024-04-18 12:48:16 UTC
Applying: Bug 29507: Tidy
error: sha1 information is lacking or useless (misc/cronjobs/automatic_renewals.pl).
error: could not build fake ancestor
Comment 13 Nick Clemens (kidclamp) 2024-04-18 13:16:39 UTC
Created attachment 165091 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 14 Nick Clemens (kidclamp) 2024-04-18 13:16:41 UTC
Created attachment 165092 [details] [review]
Bug 29507: Tidy
Comment 15 Nick Clemens (kidclamp) 2024-07-02 13:00:04 UTC
Created attachment 168362 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Comment 16 Nick Clemens (kidclamp) 2024-07-02 13:00:08 UTC
Created attachment 168363 [details] [review]
Bug 29507: Tidy
Comment 17 Martin Renvoize 2024-07-02 15:31:18 UTC
Created attachment 168381 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 18 Martin Renvoize 2024-07-02 15:31:21 UTC
Created attachment 168382 [details] [review]
Bug 29507: Tidy

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 19 Martin Renvoize 2024-07-02 15:32:07 UTC
Nice little performance enhancement.  Passing QA
Comment 20 Katrin Fischer 2024-07-05 13:17:26 UTC
How can we best document this new addition to the koha-conf.xml file?
Comment 21 Katrin Fischer 2024-07-05 13:50:20 UTC
Pushed for 24.11!

Well done everyone, thank you!
Comment 22 Jonathan Druart 2024-07-05 14:09:34 UTC
Why didn't we enqueue several background jobs instead?

Isn't the point of the workers to deal with parallelization and configuration of how to process jobs?