Bug 15104 - Batch Record Modification Performance Improvement
Summary: Batch Record Modification Performance Improvement
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: cnorthcott.work
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 7923
  Show dependency treegraph
 
Reported: 2015-11-02 00:20 UTC by cnorthcott.work
Modified: 2021-04-01 13:03 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 15104: Batch Record Modification Performance Improvement (6.46 KB, patch)
2015-11-05 01:30 UTC, cnorthcott.work
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description cnorthcott.work 2015-11-02 00:20:36 UTC
Batch Record Modification has an unreasonably long run/load time. I have attempted to improve the runtime of the Batch Record Modification tool and my enhancement reduces the time to modify a batch of records by an average of 71.44%

I discovered the extended runtime was mostly due to the serialisation and de-serialisation of success messages for every modified record. I have changed the reporting on this feature to only record and display messages for errors which improved the time of modification for x records from 233.2 seconds to 66.6 seconds, saving 166.6 seconds.
Comment 1 cnorthcott.work 2015-11-02 00:26:51 UTC
The modification test case was for 2300 records, sorry forgot to update the value.
Comment 2 cnorthcott.work 2015-11-05 01:30:44 UTC
Created attachment 44446 [details] [review]
Bug 15104: Batch Record Modification Performance Improvement

Performance improvements to speed of the Modify Records by decreasing unnecessary
progress reporting of success messages and data fetching while logging.

Test Plan:
1) Prepare NTYprof (http://wiki.koha-community.org/wiki/Profiling_with_Devel::NYTProf)
   or procure a stopwatch (stopwatch will be simpler but less accurate).
2) Ensure you have a decent number of records in your Koha system (>500) if you are
   lacking records to modify in your database you can import some from files
   found here: http://wiki.koha-community.org/wiki/Free_sample_MARC_data
3) Prepare a file with biblionumbers to modify. You can create a file containing
   all record  called ItemBarcodes.txt in the current folder using the
   following command from the machine running Koha:
   echo 'select biblionumber from biblio' | sudo koha-mysql kohadev >> BiblioNums.txt
4) Navigate to Home > Tools > MARC modification templates in the Koha Intranet.
5) Create a new template called ModRec1.
6) Add a new action using:
   Select 'Add/Update' from the drop down box and in the field(s) boxes enter '090'
   (in the first box) and 'a' (in the second box) and 'Mod1' in the value field.
   Then click 'Add Action'
7) Create a new template called ModRec2.
8) Add a new action using:
   Select 'Add/Update' from the drop down box and in the field(s) boxes enter '090'
   (in the first box) and 'a' (in the second box) and 'Mod2' in the value field.
   Then click 'Add Action'
9) Navigate to Home > Tools > Batch Modify Records in the Koha Intranet
10) Click the 'Browse...' button and select BiblioNums.txt or your equilient.
11) Select ModRec1 under Use MARC Modification Template.
12) Click the 'Continue' button and scroll to the bottom of the page.
13) If you are using a stopwatch, prepare your stopwatch so that you will
   start counting seconds from the point you click the button in the next
   step.
14) Click the 'Modify Selected Records' button (simultaneously start your
    stopwatch if using one)
15) When the page appears showing completion of the change, stop your stopwatch
    and check the time or navigate to the folder you have set NYTProf to output.
16) Record the runtime. This is the pre-optimisation time.
17) If this is the pre-patch test then you should see lots of messages like:
        "The biblio #### has successfully been modified."
    If this is the post-patch test then you should see only:
        "All records have successfully been modified!"
        and a link to do another record modification.
18) Apply this patch.
19) Repeat steps 9-12 of this testplan with the patch applied, replacing ModRec1
    in step 11 with ModRec2. This will yield the post-optimisation time. If you
    are using NYTProf you will need to apply the -d:NYTProf shebang to the file
    again.
20) Compare the pre-optimisation time and post optimisation time. The second
    post-optimisation time should be faster.
Comment 3 Jonathan Druart 2015-12-02 14:27:26 UTC
Comment on attachment 44446 [details] [review]
Bug 15104: Batch Record Modification Performance Improvement

Review of attachment 44446 [details] [review]:
-----------------------------------------------------------------

::: C4/Biblio.pm
@@ +307,4 @@
>      }
>  
>      if ( C4::Context->preference("CataloguingLog") ) {
> +        logaction( "CATALOGUING", "MODIFY", $biblionumber, "biblio BEFORE=>" . $record->as_formatted );

You are logging the modified record, not the original one.

::: tools/batch_record_modification.pl
@@ +157,4 @@
>      my ( $job );
>      if ( $runinbackground ) {
> +        $job_size = scalar( @record_ids );
> +        $job = C4::BackgroundJob->new( $sessionID, "FIXME", $ENV{SCRIPT_NAME}, $job_size );

Wrong change, your are reintroducing old lines (see bug 15005).

@@ +183,4 @@
>      my $progress = 0;
>      $dbh->{RaiseError} = 1;
>      RECORD_IDS: for my $record_id ( sort { $a <=> $b } @record_ids ) {
> +        $progress++;

There is a ++$progress later, you are incrementing twice the same variable in the same block.
Comment 4 Owen Leonard 2016-03-23 12:55:28 UTC
Based on Joubu's Comment 3 I'm guessing this should be marked as "Failed QA" for now.