| Summary: | Speed up auto renew cronjob via parallel processing | ||
|---|---|---|---|
| Product: | Koha | Reporter: | Nick Clemens (kidclamp) <nick> |
| Component: | Command-line Utilities | Assignee: | Nick Clemens (kidclamp) <nick> |
| Status: | RESOLVED FIXED | QA Contact: | Martin Renvoize (ashimema) <martin.renvoize> |
| Severity: | enhancement | ||
| Priority: | P5 - low | CC: | david, dcook, jonathan.druart, lucas, matt.blenkinsop, robin, samalau |
| Version: | Main | ||
| Hardware: | All | ||
| OS: | All | ||
| Change sponsored?: | --- | Patch complexity: | --- |
| Documentation contact: | Documentation submission: | ||
| Text to go in the release notes: |
This enhancement enables using parallel processing to speed up the running of the automatic renewals cron job (misc/cronjobs/automatic_renewals.pl). This cron job can take hours to run for libraries with thousands of items to renew.
To use this enhancement, add this setting to the instance's koha-conf.xml - adjusting the value depending on the system resources available:
<auto_renew_cronjob>
<parallel_loops_count>1</parallel_loops_count>
</auto_renew_cronjob>
|
Version(s) released in: |
24.11.00
|
| Circulation function: | |||
| Bug Depends on: | |||
| Bug Blocks: | 38156 | ||
| Attachments: |
Bug 28833: Speed up auto renew cronjob via parallel processing
Bug 29057: Speed up auto renew cronjob via parallel processing Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Tidy Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Tidy Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Tidy Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Tidy Bug 29507: Speed up auto renew cronjob via parallel processing Bug 29507: Tidy |
||
|
Description
Nick Clemens (kidclamp)
2021-11-17 17:34:35 UTC
Created attachment 127743 [details] [review] Bug 28833: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Created attachment 127744 [details] [review] Bug 29057: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Created attachment 132056 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible, it could take under 2 hours. Test Plan: 0) Install the Perl library Parallel::ForkManager 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Does not apply. I am wondering how this can work with the digest option. Created attachment 150692 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of holds in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values (In reply to Jonathan Druart from comment #4) > I am wondering how this can work with the digest option. There's a loop to split based on borrowers. We push all the issues from the first borrower to chunk 0, when we complete one borrower we push the next on to the next chunk, then the next borrower to next chunk When we have pushed to as many chunks as we need (as many loops) we start again. So a single borrower, no matter how many checkouts, will always be processed by a single fork. It means the chunks won't be necessarily the same size, but they will work with digest Created attachment 157772 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of issues in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Created attachment 161314 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of issues in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Created attachment 161315 [details] [review] Bug 29507: Tidy Created attachment 161326 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of issues in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> Created attachment 161327 [details] [review] Bug 29507: Tidy Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> Applying: Bug 29507: Tidy error: sha1 information is lacking or useless (misc/cronjobs/automatic_renewals.pl). error: could not build fake ancestor Created attachment 165091 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of issues in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> Created attachment 165092 [details] [review] Bug 29507: Tidy Created attachment 168362 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of issues in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> Created attachment 168363 [details] [review] Bug 29507: Tidy Created attachment 168381 [details] [review] Bug 29507: Speed up auto renew cronjob via parallel processing The cron can take a very long time to run on systems with many issues. For example, a partner with ~250k auto_renew issues is taking about 9 hours to run. If we run that same number of issues in 5 parallel chunks ( splitting the number of issues as evenly as possible ), it could take under 2 hours. Test Plan: 1) Generate a number of issues marked for auto_renew 2) Run the automatic_renewals.pl, use the `time` utility to track how much time it took to run 3) Set parallel_loops to 10 in auto_renew_cronjob section of config in koha-conf 4) Repeat step 2, note the improvement in speed 5) Experiment with other values Signed-off-by: Matt Blenkinsop <matt.blenkinsop@ptfs-europe.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 168382 [details] [review] Bug 29507: Tidy Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Nice little performance enhancement. Passing QA How can we best document this new addition to the koha-conf.xml file? Pushed for 24.11! Well done everyone, thank you! Why didn't we enqueue several background jobs instead? Isn't the point of the workers to deal with parallelization and configuration of how to process jobs? (In reply to Jonathan Druart from comment #22) > Why didn't we enqueue several background jobs instead? > > Isn't the point of the workers to deal with parallelization and > configuration of how to process jobs? I think we didn't have as much progress on the background jobs when I started this. Would it be the goal now to have most cronjobs submit background jobs? (In reply to Nick Clemens (kidclamp) from comment #23) > (In reply to Jonathan Druart from comment #22) > > Why didn't we enqueue several background jobs instead? > > > > Isn't the point of the workers to deal with parallelization and > > configuration of how to process jobs? > > I think we didn't have as much progress on the background jobs when I > started this. Would it be the goal now to have most cronjobs submit > background jobs? I see pros and cons with both approaches I think. If we did enqueue them as background jobs and there were a large number of jobs across multiple Koha instances, that would produce a higher load on the server than doing the work in the cronjob itself. While parallel processing in the cronjob would produce a higher load as well, it would still be smaller than the load using background jobs (although the impact will vary depending on the number of instances per system and the amount of resources available) But... we'd get through the cronjobs faster. (We have found the cronjobs can take a very long time to finish, because each cronjob has to run for each library one by one.) I'm intrigued by both approaches... I've had a go at adding a release note - please update if I haven't described how to use this enhancement correctly! |