Attachment 72481 Details for Bug 20342 – Bug 20342: Fix table locking issue, progress bar not updating, and allow use of plack for Stage MARC Import

[patch] Bug 20342: Fix table locking issue, progress bar not updating, and allow use of plack for Stage MARC Import

Bug-20342-Fix-table-locking-issue-progress-bar-not.patch (text/plain), 4.60 KB, created by Kyle M Hall (khall) on 2018-03-06 14:54:12 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Kyle M Hall (khall)

Created: 2018-03-06 14:54:12 UTC

Size: 4.60 KB

patch

obsolete

>From 5f6cdfbf1c4378940a07417a6da32a98d85be84a Mon Sep 17 00:00:00 2001
>From: Kyle M Hall <kyle@bywatersolutions.com>
>Date: Tue, 6 Mar 2018 10:38:38 +0000
>Subject: [PATCH] Bug 20342: Fix table locking issue, progress bar not
> updating, and allow use of plack for Stage MARC Import
>
>Have you ever seen this error?
>DBD::mysql::st execute failed: Deadlock found when trying to get lock; try restarting transaction [for Statement "INSERT INTO import_record_matches (import_record_id, candidate_match_id, score),
>
>Have you ever noticed your progress bar during stage for import jumps from 0% to 100% with no updates in between?
>
>Would you like stage-marc-import.pl to work under plack?
>
>If you answered yes to any of the following questions, this bug is for you!
>
>During uploads of large records using matching rules, we were getting 'Failed to submit form' popups on the marc record importer.
>
>First, I noticed that the parent process of stage-marc-import.pl wasn't returning the job id via ajax until the child process had completed. This works fine for smaller files as they process fast. For large files ( and in particular when using matching rules ) the process takes *much* longer.
>
>It appears that $.ajax will resubmit the form if it doesn't get a response back in a given time ( I would estimate that to be about 30 seconds or so ). This causes a *second* instance of stage-marc-import.pl to be run. So now we have two processes both running transactions on the same table for the same batch id. I believe this is what is causing the Deadlock.
>
>So, the crux of the matter is that for whatever reason, the parent process will not output until the child process has gone away. Hours of research entailed. The final answer is that the child process must be detached from the parent process. I tried doing this via setsid() but it did not help. Finally, I found that using Net::Server::Daemonize did!
>
>After I had my solution, I decided to see if stage-marc-import.pl and background-job-progress.pl could now be run under plack, and it appears that they can!
>
>Test Plan:
>1) Apply this patch
>2) Update /etc/koha/apache-shared-intranet-plack.conf manually, comment
>   out the lines for stage-marc-import.pl and  background-job-progress.pl
>3) Restart apache, plack and memcached
>4) Stage a large record in stage-marc-import.pl, use a matching rule
>5) Note it works!
>---
> debian/templates/apache-shared-intranet-plack.conf | 2 --
> tools/stage-marc-import.pl                         | 8 +++++++-
> 2 files changed, 7 insertions(+), 3 deletions(-)
>
>diff --git a/debian/templates/apache-shared-intranet-plack.conf b/debian/templates/apache-shared-intranet-plack.conf
>index 5ae8e07..6ce25cd 100644
>--- a/debian/templates/apache-shared-intranet-plack.conf
>+++ b/debian/templates/apache-shared-intranet-plack.conf
>@@ -12,11 +12,9 @@
>         #Â FIXME: These scripts should be fixed so they
>         #Â don't break under plack/starman
>         ProxyPass "/cgi-bin/koha/offline_circ/process_koc.pl" "!"
>-        ProxyPass "/cgi-bin/koha/tools/background-job-progress.pl" "!"
>         ProxyPass "/cgi-bin/koha/tools/batch_record_modification.pl" "!"
>         ProxyPass "/cgi-bin/koha/tools/batchMod.pl" "!"
>         ProxyPass "/cgi-bin/koha/tools/manage-marc-import.pl" "!"
>-        ProxyPass "/cgi-bin/koha/tools/stage-marc-import.pl" "!"
>         ProxyPass "/cgi-bin/koha/tools/upload-cover-image.pl" "!"
>         ProxyPass "/cgi-bin/koha/svc/cataloguing/metasearch" "!"
> 
>diff --git a/tools/stage-marc-import.pl b/tools/stage-marc-import.pl
>index f2acc58..ee698a0 100755
>--- a/tools/stage-marc-import.pl
>+++ b/tools/stage-marc-import.pl
>@@ -30,6 +30,7 @@ use Modern::Perl;
> use CGI qw ( -utf8 );
> use CGI::Cookie;
> use MARC::File::USMARC;
>+use Net::Server::Daemonize qw(daemonize);
> 
> # Koha modules used
> use C4::Context;
>@@ -118,10 +119,15 @@ if ($completedJobID) {
>             print '{"jobID":"' . $jobID . '"}';
>             exit 0;
>         } elsif (defined $pid) {
>+            # Daemonize the child process os the parent will print its output
>+            my $pidfile = "/tmp/async$$.pid"; # deamonize requires a pid file even though it is supposed to be optional
>+            my @current_user = getpwuid($<);
>+            daemonize( $current_user[2], $current_user[3], $pidfile );
>+            unlink($pidfile);    # otherwise it doesn't go away
>             # child
>             # close STDOUT to signal to Apache that
>             # we're now running in the background
>-            close STDOUT;
>+            close STDOUT; # FIXME I think this is no longer necessary
>             # close STDERR; # there is no good reason to close STDERR
>         } else {
>             # fork failed, so exit immediately
>-- 
>2.10.2

Actions: View | Diff | Splinter Review

Attachments on bug 20342: 72481 | 72483