29507 – Speed up auto renew cronjob via parallel processing

Bug 29507 - Speed up auto renew cronjob via parallel processing

Summary: Speed up auto renew cronjob via parallel processing

Status:	RESOLVED FIXED

Alias:	None

Product:	Koha
Classification:	Unclassified
Component:	Command-line Utilities (show other bugs)
Version:	Main
Hardware:	All All

Importance:	P5 - low enhancement
Assignee:	Nick Clemens (kidclamp)
QA Contact:	Martin Renvoize (ashimema)

URL:
Keywords:

Depends on:
Blocks:	38156
	Show dependency tree / graph

Reported:	2021-11-17 17:34 UTC by Nick Clemens (kidclamp)
Modified:	2024-11-18 17:40 UTC (History)
CC List:	7 users (show)

See Also:
Change sponsored?:	---
Patch complexity:	---
Documentation contact:
Documentation submission:
Text to go in the release notes:	This enhancement enables using parallel processing to speed up the running of the automatic renewals cron job (misc/cronjobs/automatic_renewals.pl). This cron job can take hours to run for libraries with thousands of items to renew. To use this enhancement, add this setting to the instance's koha-conf.xml - adjusting the value depending on the system resources available: <auto_renew_cronjob> <parallel_loops_count>1</parallel_loops_count> </auto_renew_cronjob>
Version(s) released in:	24.11.00
Circulation function:

Attachments
Bug 28833: Speed up auto renew cronjob via parallel processing (15.17 KB, patch) 2021-11-17 17:35 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29057: Speed up auto renew cronjob via parallel processing (15.17 KB, patch) 2021-11-17 17:45 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (15.04 KB, patch) 2022-03-23 15:17 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (18.06 KB, patch) 2023-05-04 19:29 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (18.63 KB, patch) 2023-10-25 11:20 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.59 KB, patch) 2024-01-23 19:11 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Tidy (20.12 KB, patch) 2024-01-23 19:11 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.68 KB, patch) 2024-01-24 11:33 UTC, Matt Blenkinsop	Details \| Diff \| Splinter Review
Bug 29507: Tidy (20.21 KB, patch) 2024-01-24 11:33 UTC, Matt Blenkinsop	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.66 KB, patch) 2024-04-18 13:16 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Tidy (20.24 KB, patch) 2024-04-18 13:16 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.66 KB, patch) 2024-07-02 13:00 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Tidy (20.35 KB, patch) 2024-07-02 13:00 UTC, Nick Clemens (kidclamp)	Details \| Diff \| Splinter Review
Bug 29507: Speed up auto renew cronjob via parallel processing (4.72 KB, patch) 2024-07-02 15:31 UTC, Martin Renvoize (ashimema)	Details \| Diff \| Splinter Review
Bug 29507: Tidy (20.41 KB, patch) 2024-07-02 15:31 UTC, Martin Renvoize (ashimema)	Details \| Diff \| Splinter Review
Show Obsolete (13) View All Add an attachment (proposed patch, testcase, etc.)

Note You need to log in before you can comment on or make changes to this bug.

Description Nick Clemens (kidclamp) 2021-11-17 17:34:35 UTC

Comment 1 Nick Clemens (kidclamp) 2021-11-17 17:35:06 UTC Comment hidden (obsolete)

Created attachment 127743 [details] [review]
Bug 28833: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Comment 2 Nick Clemens (kidclamp) 2021-11-17 17:45:32 UTC

Created attachment 127744 [details] [review]
Bug 29057: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Comment 3 Nick Clemens (kidclamp) 2022-03-23 15:17:48 UTC

Created attachment 132056 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours.

Test Plan:
0) Install the Perl library Parallel::ForkManager
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Comment 4 Jonathan Druart 2023-05-04 13:19:58 UTC

Does not apply.

I am wondering how this can work with the digest option.

Comment 5 Nick Clemens (kidclamp) 2023-05-04 19:29:40 UTC

Created attachment 150692 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of holds in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Comment 6 Nick Clemens (kidclamp) 2023-05-04 19:34:59 UTC

(In reply to Jonathan Druart from comment #4)

> I am wondering how this can work with the digest option.

There's a loop to split based on borrowers. We push all the issues from the first borrower to chunk 0, when we complete one borrower we push the next on to the next chunk, then the next borrower to next chunk

When we have pushed to as many chunks as we need (as many loops) we start again.

So a single borrower, no matter how many checkouts, will always be processed by a single fork. It means the chunks won't be necessarily the same size, but they will work with digest

Comment 7 Nick Clemens (kidclamp) 2023-10-25 11:20:46 UTC

Created attachment 157772 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Comment 8 Nick Clemens (kidclamp) 2024-01-23 19:11:16 UTC

Created attachment 161314 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Comment 9 Nick Clemens (kidclamp) 2024-01-23 19:11:18 UTC

Created attachment 161315 [details] [review]
Bug 29507: Tidy

Comment 10 Matt Blenkinsop 2024-01-24 11:33:52 UTC

Created attachment 161326 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>

Comment 11 Matt Blenkinsop 2024-01-24 11:33:55 UTC

Created attachment 161327 [details] [review]
Bug 29507: Tidy

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>

Comment 12 Marcel de Rooy 2024-04-18 12:48:16 UTC

Applying: Bug 29507: Tidy
error: sha1 information is lacking or useless (misc/cronjobs/automatic_renewals.pl).
error: could not build fake ancestor

Comment 13 Nick Clemens (kidclamp) 2024-04-18 13:16:39 UTC

Created attachment 165091 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>

Comment 14 Nick Clemens (kidclamp) 2024-04-18 13:16:41 UTC

Created attachment 165092 [details] [review]
Bug 29507: Tidy

Comment 15 Nick Clemens (kidclamp) 2024-07-02 13:00:04 UTC

Created attachment 168362 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>

Comment 16 Nick Clemens (kidclamp) 2024-07-02 13:00:08 UTC

Created attachment 168363 [details] [review]
Bug 29507: Tidy

Comment 17 Martin Renvoize (ashimema) 2024-07-02 15:31:18 UTC

Created attachment 168381 [details] [review]
Bug 29507: Speed up auto renew cronjob via parallel processing

The cron can take a very long time to run on systems with many issues.
For example, a partner with ~250k auto_renew issues is taking about 9 hours to run.

If we run that same number of issues in 5 parallel chunks
( splitting the number of issues as evenly as possible ), it could take under 2 hours.

Test Plan:
1) Generate a number of issues marked for auto_renew
2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run
3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf
4) Repeat step 2, note the improvement in speed
5) Experiment with other values

Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Comment 18 Martin Renvoize (ashimema) 2024-07-02 15:31:21 UTC

Created attachment 168382 [details] [review]
Bug 29507: Tidy

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Comment 19 Martin Renvoize (ashimema) 2024-07-02 15:32:07 UTC

Nice little performance enhancement.  Passing QA

Comment 20 Katrin Fischer 2024-07-05 13:17:26 UTC

How can we best document this new addition to the koha-conf.xml file?

Comment 21 Katrin Fischer 2024-07-05 13:50:20 UTC

Pushed for 24.11!

Well done everyone, thank you!

Comment 22 Jonathan Druart 2024-07-05 14:09:34 UTC

Why didn't we enqueue several background jobs instead?

Isn't the point of the workers to deal with parallelization and configuration of how to process jobs?

Comment 23 Nick Clemens (kidclamp) 2024-07-22 13:40:33 UTC

(In reply to Jonathan Druart from comment #22)
> Why didn't we enqueue several background jobs instead?
> 
> Isn't the point of the workers to deal with parallelization and
> configuration of how to process jobs?

I think we didn't have as much progress on the background jobs when I started this. Would it be the goal now to have most cronjobs submit background jobs?

Comment 24 David Cook 2024-07-22 23:53:05 UTC

(In reply to Nick Clemens (kidclamp) from comment #23)
> (In reply to Jonathan Druart from comment #22)
> > Why didn't we enqueue several background jobs instead?
> > 
> > Isn't the point of the workers to deal with parallelization and
> > configuration of how to process jobs?
> 
> I think we didn't have as much progress on the background jobs when I
> started this. Would it be the goal now to have most cronjobs submit
> background jobs?

I see pros and cons with both approaches I think.

If we did enqueue them as background jobs and there were a large number of jobs across multiple Koha instances, that would produce a higher load on the server than doing the work in the cronjob itself.

While parallel processing in the cronjob would produce a higher load as well, it would still be smaller than the load using background jobs (although the impact will vary depending on the number of instances per system and the amount of resources available)

But... we'd get through the cronjobs faster. (We have found the cronjobs can take a very long time to finish, because each cronjob has to run for each library one by one.)

I'm intrigued by both approaches...

Comment 25 David Nind 2024-07-23 03:33:36 UTC

I've had a go at adding a release note - please update if I haven't described how to use this enhancement correctly!