Bugzilla – Attachment 95446 Details for
Bug 24027
Adding multiple items is slow
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 24027: ModZebra should be called once after all items added, not on each item add
Bug-24027-ModZebra-should-be-called-once-after-all.patch (text/plain), 5.93 KB, created by
Andrii Nugged
on 2019-11-14 18:07:08 UTC
(
hide
)
Description:
Bug 24027: ModZebra should be called once after all items added, not on each item add
Filename:
MIME Type:
Creator:
Andrii Nugged
Created:
2019-11-14 18:07:08 UTC
Size:
5.93 KB
patch
obsolete
>From a9d6bae6de3872318e89ff32311766f14a14e6ec Mon Sep 17 00:00:00 2001 >From: Andrew Nugged <nugged@gmail.com> >Date: Thu, 14 Nov 2019 17:59:54 +0000 >Subject: [PATCH] Bug 24027: ModZebra should be called once after all items > added, not on each item add > >Why happened: >- call to ModZebra was done after EACH item added, > but it was called only with main biblionumber only, > so call was the same on each of requests >- and also time spent in that ModZebra sub increased with every > next hundred items in DB for that element: so adding every next 100 > was slower and slower, > >Solved: >- now it's called only once (by adding some extra parameter to "AddItem*" sub set in postponed mode) >- and now adding of elements not so heavily depends from how much items was in DB before. > >Test plan / How to replicate the issue: >(test depends from how already populated DB and Zebra/Elasticsearch DBs > so might be not so noticeably fast on empty DBs) > >- go to one of bibliotitems cataloguing/additem.pl?biblionumber=XXX pages >- press button "add multiple copies of this item", put 1000 on slower machines, 5000 on faster >- start measuring time + submit page/form > >It takes some amount of time even up to timeout. >After applying patch, run the same procedure for same number of items. >Note: it goes fast in both variants on empty database so time not so >noticeable (also depents how ModZebra-related stuff is configured). > >Also it slows more linearly when number of items grows. > >But, again, it heavily depends how database is populated and indexes are rebuild >but it's obvious that it will me much more efficient to call "ModZebra" once >after 1000 addings, then call it 1000 times on each item created in the loop. >--- > C4/Items.pm | 25 +++++++++++++++++++------ > cataloguing/additem.pl | 6 +++++- > 2 files changed, 24 insertions(+), 7 deletions(-) > >diff --git a/C4/Items.pm b/C4/Items.pm >index b81fb0aa17..5128984ab5 100644 >--- a/C4/Items.pm >+++ b/C4/Items.pm >@@ -144,7 +144,10 @@ sub CartToShelf { > =head2 AddItemFromMarc > > my ($biblionumber, $biblioitemnumber, $itemnumber) >- = AddItemFromMarc($source_item_marc, $biblionumber); >+ = AddItemFromMarc($source_item_marc, $biblionumber[, $hash_params]); >+ >+$hash_params: >+ postpone_indexes_update => true / false > > Given a MARC::Record object containing an embedded item > record and a biblionumber, create a new item record. >@@ -152,7 +155,10 @@ record and a biblionumber, create a new item record. > =cut > > sub AddItemFromMarc { >- my ( $source_item_marc, $biblionumber ) = @_; >+ my $source_item_marc = shift; >+ my $biblionumber = shift; >+ my $hash_params = @_ ? shift : {}; >+ > my $dbh = C4::Context->dbh; > > # parse item hash from MARC >@@ -163,13 +169,13 @@ sub AddItemFromMarc { > $localitemmarc->append_fields( $source_item_marc->field($itemtag) ); > my $item = C4::Biblio::TransformMarcToKoha( $localitemmarc, $frameworkcode, 'items' ); > my $unlinked_item_subfields = _get_unlinked_item_subfields( $localitemmarc, $frameworkcode ); >- return AddItem( $item, $biblionumber, $dbh, $frameworkcode, $unlinked_item_subfields ); >+ return AddItem( $item, $biblionumber, $dbh, $frameworkcode, $unlinked_item_subfields, $hash_params ); > } > > =head2 AddItem > > my ($biblionumber, $biblioitemnumber, $itemnumber) >- = AddItem($item, $biblionumber[, $dbh, $frameworkcode, $unlinked_item_subfields]); >+ = AddItem($item, $biblionumber[, $dbh, $frameworkcode, $unlinked_item_subfields, $hash_params]); > > Given a hash containing item column names as keys, > create a new Koha item record. >@@ -178,13 +184,18 @@ The first two optional parameters (C<$dbh> and C<$frameworkcode>) > do not need to be supplied for general use; they exist > simply to allow them to be picked up from AddItemFromMarc. > >-The final optional parameter, C<$unlinked_item_subfields>, contains >+Optional parameter, C<$unlinked_item_subfields>, contains > an arrayref containing subfields present in the original MARC > representation of the item (e.g., from the item editor) that are > not mapped to C<items> columns directly but should instead > be stored in C<items.more_subfields_xml> and included in > the biblio items tag for display and indexing. > >+The final optional parameter, C<$hash_params>, for now expected to contain >+'postpone_indexes_update' key, which when true prevents calling of ModZebra sub >+so this used when adding items in a batch and then ModZebra called in >+C<additem.pl> after the whole loop. >+ > =cut > > sub AddItem { >@@ -197,6 +208,7 @@ sub AddItem { > if (@_) { > $unlinked_item_subfields = shift; > } >+ my $hash_params = @_ ? shift : {}; > > # needs old biblionumber and biblioitemnumber > $item->{'biblionumber'} = $biblionumber; >@@ -220,7 +232,8 @@ sub AddItem { > > $item->{'itemnumber'} = $itemnumber; > >- C4::Biblio::ModZebra( $item->{biblionumber}, "specialUpdate", "biblioserver" ); >+ C4::Biblio::ModZebra( $item->{biblionumber}, "specialUpdate", "biblioserver" ) >+ unless $hash_params->{postpone_indexes_update}; > > logaction( "CATALOGUING", "ADD", $itemnumber, "item" ) > if C4::Context->preference("CataloguingLog"); >diff --git a/cataloguing/additem.pl b/cataloguing/additem.pl >index 2c371def28..f0fcf33ecc 100755 >--- a/cataloguing/additem.pl >+++ b/cataloguing/additem.pl >@@ -600,7 +600,8 @@ if ($op eq "additem") { > > # Adding the item > if (!$exist_itemnumber) { >- my ($oldbiblionumber,$oldbibnum,$oldbibitemnum) = AddItemFromMarc($record,$biblionumber); >+ my ( $oldbiblionumber, $oldbibnum, $oldbibitemnum ) = >+ AddItemFromMarc( $record, $biblionumber, { postpone_indexes_update => 1 } ); > set_item_default_location($oldbibitemnum); > > # We count the item only if it was really added >@@ -614,6 +615,9 @@ if ($op eq "additem") { > # Preparing the next iteration > $oldbarcode = $barcodevalue; > } >+ >+ C4::Biblio::ModZebra( $biblionumber, "specialUpdate", "biblioserver" ); >+ > undef($itemrecord); > } > } >-- >2.17.1
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 24027
:
95413
|
95414
|
95417
|
95446
|
95473
|
95663
|
98250
|
98251
|
98252
|
98253
|
102464
|
102474
|
102567
|
102568
|
102572
|
102578
|
102579
|
102580