Bugzilla – Attachment 73166 Details for
Bug 20377
Add transaction and skip_merge to 'remove_unused_authorities.pl' script
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 20377: Various speed tweaks to 'remove_unused_authorities.pl' script
Bug-20377-Various-speed-tweaks-to-removeunusedauth.patch (text/plain), 3.12 KB, created by
Marcel de Rooy
on 2018-03-23 09:42:04 UTC
(
hide
)
Description:
Bug 20377: Various speed tweaks to 'remove_unused_authorities.pl' script
Filename:
MIME Type:
Creator:
Marcel de Rooy
Created:
2018-03-23 09:42:04 UTC
Size:
3.12 KB
patch
obsolete
>From bc72f8ab19c249fae8e527bc0bc3a317808eb927 Mon Sep 17 00:00:00 2001 >From: Mason James <mtj@kohaaloha.com> >Date: Tue, 13 Mar 2018 11:36:23 +1300 >Subject: [PATCH] Bug 20377: Various speed tweaks to > 'remove_unused_authorities.pl' script >Content-Type: text/plain; charset=utf-8 > >Here is a patch to make 'remove_unused_authorities.pl' a bit faster >i get a 5x speedup after this patch > >fyi: i pinched most of this patch from bulkmarcimport.pl > >1/ import a bunch of authority record into a Koha, and index > >2/ run script, note execution time > $ time perl ./misc/migration_tools/remove_unused_authorities.pl > real 0m50s > >3/ apply patch > >4/ run script again, and notice it's a bit faster > $ time perl ./misc/migration_tools/remove_unused_authorities.pl > real 0m10s > >Signed-off-by: Brendan Gallagher <brendan@bywatersolutions.com> > >Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> >--- > misc/migration_tools/remove_unused_authorities.pl | 21 +++++++++++++++++++-- > 1 file changed, 19 insertions(+), 2 deletions(-) > >diff --git a/misc/migration_tools/remove_unused_authorities.pl b/misc/migration_tools/remove_unused_authorities.pl >index f4e0ecb..9852229 100755 >--- a/misc/migration_tools/remove_unused_authorities.pl >+++ b/misc/migration_tools/remove_unused_authorities.pl >@@ -32,12 +32,18 @@ use Koha::SearchEngine::Search; > my @authtypes; > my $want_help = 0; > my $test = 0; >+my $commit; >+my $i = 0; >+ > GetOptions( > 'aut|authtypecode:s' => \@authtypes, > 't|test' => \$test, >+ 'c|commit:i' => \$commit, > 'h|help' => \$want_help > ); > >+my $commitnum = $commit ? $commit : 50; >+ > if ($want_help) { > print_usage(); > exit 0; >@@ -63,8 +69,9 @@ my $rqsql = "SELECT * from auth_header where 1"; > $rqsql .= " AND authtypecode IN (".join(",",map{$dbh->quote($_)}@authtypes).")" if @authtypes; > my $rqselect = $dbh->prepare($rqsql); > $|=1; >- > $rqselect->execute; >+ >+$dbh->{AutoCommit} = 0; > my $counter=0; > my $totdeleted=0; > my $totundeleted=0; >@@ -82,13 +89,22 @@ while (my $data=$rqselect->fetchrow_hashref){ > print "$counter\n" unless $counter++ % 100; > # if found, delete, otherwise, just count > if ($used>=$thresholdmin and $used<=$thresholdmax){ >- DelAuthority({ authid => $data->{'authid'} }) unless $test; >+ >+ # call with 'skip_merge' arg, as we already know these authority records >+ # are not linked to any bibs >+ DelAuthority({ authid => $data->{'authid'}, skip_merge => 1 }) unless $test; >+ > $totdeleted++; > } else { > $totundeleted++; > } >+ $i++; >+ $dbh->commit() if (0 == $i % $commitnum); > } > >+$dbh->commit(); >+$dbh->{AutoCommit} = 1; >+ > print "$counter authorities parsed, $totdeleted deleted and $totundeleted unchanged because used\n"; > > >@@ -110,6 +126,7 @@ operator to confirm the deletion of each authority record. > parameters > --aut|authtypecode TYPE the list of authtypes to check > --test or -t test mode, don't delete really, just count >+ --commit or -c the number of records to wait before performing a 'commit' operation > --help or -h show this message. > > _USAGE_ >-- >2.1.4
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 20377
:
72727
|
72775
| 73166