Attachment 171490 Details for Bug 37775 – Bug 37775: Remove delayed commits from update_totalissues.pl and add progress option

[patch] Bug 37775: Remove delayed commits from update_totalissues.pl and add progress option

Bug-37775-Remove-delayed-commits-from-updatetotali.patch (text/plain), 3.53 KB, created by Brendan Lawlor on 2024-09-13 17:58:27 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Brendan Lawlor

Created: 2024-09-13 17:58:27 UTC

Size: 3.53 KB

patch

obsolete

>From e6aa8ac8c8100200a662398bbb5f5c593cfc5d45 Mon Sep 17 00:00:00 2001
>From: Nick Clemens <nick@bywatersolutions.com>
>Date: Fri, 30 Aug 2024 12:47:07 +0000
>Subject: [PATCH] Bug 37775: Remove delayed commits from update_totalissues.pl
> and add progress option
>
>This patch removes setting AutoCommit to 0 and commiting only every X records.
>Instead we commit as we go and report progress using a parameter.
>
>Bug 36474 reduced the numebr of changes that are being committed, so this should be a reasonable change. The
>use of commits without transactions was causing problems if the library was active while the script ran.
>
>To test:
>1 - perl misc/cronjobs/update_totalissues.pl -c
>2 - Script runs, but with unknown parameter
>3 - perl misc/cronjobs/update_totalissues.pl -p 10
>4 - Script runs and reports every 10 records
>5 - per; misc/cronjobs/update_totalissues.pl
>6 - Script runs and reports every 100 records by default
>
>Signed-off-by: Brendan Lawlor <blawlor@clamsnet.org>
>---
> misc/cronjobs/update_totalissues.pl | 24 +++++++++---------------
> 1 file changed, 9 insertions(+), 15 deletions(-)
>
>diff --git a/misc/cronjobs/update_totalissues.pl b/misc/cronjobs/update_totalissues.pl
>index a5f7a5f528..43d1a46ef4 100755
>--- a/misc/cronjobs/update_totalissues.pl
>+++ b/misc/cronjobs/update_totalissues.pl
>@@ -49,7 +49,7 @@ my $interval;
> my $usestats    = 0;
> my $useitems    = 0;
> my $incremental = 0;
>-my $commit      = 100;
>+my $progress      = 100;
> my $unit;
> 
> my $command_line_options = join(" ",@ARGV);
>@@ -62,7 +62,7 @@ my $result = GetOptions(
>     'use-stats'    => \$usestats,
>     'use-items'    => \$useitems,
>     'incremental'  => \$incremental,
>-    'c|commit=i'   => \$commit,
>+    'p|progress=i'   => \$progress,
>     'h|help'       => \$want_help
> );
> 
>@@ -88,14 +88,11 @@ unless ( $usestats || $useitems ) {
>     $want_help = 1;
> }
> 
>-if ( not $result or $want_help ) {
>-    usage();
>-}
>+usage() if $want_help;
> 
> cronlogaction({ info => $command_line_options });
> 
> my $dbh = C4::Context->dbh;
>-$dbh->{AutoCommit} = 0;
> 
> my $num_bibs_processed = 0;
> my $num_bibs_updated   = 0;
>@@ -146,7 +143,6 @@ sub process_stats {
>  GROUP BY biblio.biblionumber";
>     process_query( $query, $limit );
> 
>-    $dbh->commit();
> }
> 
> sub process_query {
>@@ -180,12 +176,11 @@ sub process_query {
>                 $num_bibs_updated++;
>             }
>         }
>-        if ( not $test_only and ( $num_bibs_processed % $commit ) == 0 ) {
>-            print_progress_and_commit($num_bibs_processed);
>+        if ( not $test_only and ( $num_bibs_processed % $progress ) == 0 ) {
>+            print_progress($num_bibs_processed);
>         }
>     }
> 
>-    $dbh->commit();
> }
> 
> sub report {
>@@ -209,9 +204,8 @@ _SUMMARY_
>     print $summary;
> }
> 
>-sub print_progress_and_commit {
>+sub print_progress {
>     my $recs = shift;
>-    $dbh->commit();
>     print "... processed $recs records\n";
> }
> 
>@@ -223,7 +217,7 @@ update_totalissues.pl
> 
>   update_totalissues.pl --use-stats
>   update_totalissues.pl --use-items
>-  update_totalissues.pl --commit=1000
>+  update_totalissues.pl --progress=1000
>   update_totalissues.pl --since='2012-01-01'
>   update_totalissues.pl --interval=30d
> 
>@@ -273,9 +267,9 @@ job to update popularity information during low-usage periods. If neither
> --since or --interval are specified, incremental mode will default to
> processing the last twenty-four hours.
> 
>-=item B<--commit=N>
>+=item B<--progress=N>
> 
>-Commit the results to the database after every N records are processed.
>+Print the progress to standart output after every N records are processed.
> 
> =item B<--test>
> 
>-- 
>2.39.2

Actions: View | Diff | Splinter Review

Attachments on bug 37775: 170908 | 170909 | 171490 | 171491 | 171648 | 171649