Bug 5332 - Add batch reversion capability to bin/commit_biblios_file.pl
Summary: Add batch reversion capability to bin/commit_biblios_file.pl
Status: RESOLVED DUPLICATE of bug 2060
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: rel_3_2
Hardware: All All
: P3 enhancement (vote)
Assignee: Galen Charlton
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2010-10-31 11:22 UTC by Doug Kingston
Modified: 2012-08-10 10:04 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
version with new functionality (4.63 KB, text/x-perl-script)
2010-10-31 11:22 UTC, Doug Kingston
Details
Proposed patch (4.26 KB, patch)
2010-10-31 12:53 UTC, Doug Kingston
Details | Diff | Splinter Review
Wrong file - please ignore (243.17 KB, application/octet-stream)
2010-11-13 22:55 UTC, Doug Kingston
Details
Gzip of MARC import file with bad records (237.67 KB, application/octet-stream)
2010-11-14 10:25 UTC, Doug Kingston
Details
Add batch revert capability (--revert) to commit_biblios_file.pl. Also add --progress-interval to facilitate debugging bad MARC batches. (4.33 KB, patch)
2012-06-11 15:32 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Doug Kingston 2010-10-31 11:22:00 UTC
Created attachment 2724 [details]
version with new functionality

commit_biblios_file.pl needs a way to revert a batch in bulk.  I have done this code and will attache it to bug for review and potential inclusion.

The basis for this enhancement was the a bad batch of MARC records that had some unknown Unicode characters in them which cause the import function from the web interface to fail after importing a portion of the records.  At that point you are completely stuck.  You can't complete the import due to the errors and there is no way to *revert* either.  You are stuck with a half imported batch.

My change is to add a --revert flag to commit_biblios_file.pl which provides an
interface to the pre-existing BatchRevertBibRecords function.

I have also added a --progress-interval flag to allow the default commit interval to be changed to aid in debugging bad import batches.
Comment 1 Doug Kingston 2010-10-31 12:53:16 UTC Comment hidden (obsolete)
Comment 2 Chris Cormack 2010-11-08 02:38:10 UTC
Pushed to new/enh/bug_5332 along with Galens follow up patches, please test and report so I can merge to master
Comment 3 Doug Kingston 2010-11-10 23:01:10 UTC
(In reply to comment #2)
> Pushed to new/enh/bug_5332 along with Galens follow up patches, please test and
> report so I can merge to master

I'm new to this process.  Do you need me to test this?  If so, I'm not sure how to work with the referenc new/enh/bug_5332.  Pointers welcome.
Comment 4 Chris Cormack 2010-11-12 02:42:44 UTC
Doug, are you running a development server or a test server anywhere?

If so, and if it is running from git, you can do this

git checkout -b bug_5332 origin/new/enh/bug_5332

And you will now be running that branch.  If you aren't thats fine, just let us know and we will get someone else to test.
Comment 5 Doug Kingston 2010-11-12 23:47:50 UTC
(In reply to comment #4)
> Doug, are you running a development server or a test server anywhere?
> 
> If so, and if it is running from git, you can do this
> 
> git checkout -b bug_5332 origin/new/enh/bug_5332
> 
> And you will now be running that branch.  If you aren't thats fine, just let us
> know and we will get someone else to test.

Sorry, at the moment I am running a stock 3.2.  We don't have a QA environment yet.  I'll try to get more organized and git-aware as I make more changes, but for now I think it will be faster for someone else.  Do you need a duff marc import file (with some broken unicode in it) to create a broken import?
Comment 6 Chris Cormack 2010-11-13 20:21:21 UTC
Yes please a attach a file that breaks it currently, and we can use that to test that it is now fixed.
Comment 7 Doug Kingston 2010-11-13 22:55:43 UTC Comment hidden (obsolete)
Comment 8 Doug Kingston 2010-11-14 10:25:58 UTC
Created attachment 2748 [details]
Gzip of MARC import file with bad records

This gzip'd MARC21 import file has some bad unicode characters in it that will cause the import to abort part way through.  At that point the batch import is stuck.  It can't complete due to the errors and cannot be reverted from the web UI since it has not completed.
(take8.marc.gz replaces take7.marc.gz which was erroneously uploaded yesterday.)
Comment 9 Doug Kingston 2010-11-16 22:32:15 UTC
Now I am not sure which one was the problem import file (take7 - now marked obsolete, or take 8).  But its one of them.  I will try to get a test system up here so I can make a definitive statement.
Comment 10 Chris Cormack 2011-01-09 20:19:32 UTC
Still awaiting testing
Comment 11 Katrin Fischer 2012-06-11 15:32:09 UTC
Created attachment 10232 [details] [review]
Add batch revert capability (--revert) to commit_biblios_file.pl. Also add --progress-interval to facilitate debugging bad MARC batches.

The basis for this enhancement was a bad batch of MARC records that
had some unknown Unicode characters in them which caused the batch
import function from the web interface to abort after importing only
a portion of the records.  At this point you are completely stuck using
the web user interface.  You can't go forward because the only option
is to try the import again from the beginning (which fails again).
You cannot revert either since there is no way to revert a partially
committed batch.

This change uses the existing BatchRevertBibRecords function.
Bug reference: 5332

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Rebased to current master 2012-06-11 - please test.
Comment 12 Jared Camins-Esakov 2012-07-02 18:02:14 UTC
This functionality is also added by bug 2060. I overlooked this patch, and wrote more-or-less identical code when I needed the ability to revert from the command line to test authority import.
Comment 13 Magnus Enger 2012-08-10 10:04:52 UTC
The main functionality of this patch (the --revert option) is covered by the patch for Bug 2060 (which is already signed off), as pointed out by Jared in Comment 12. I'm marking this bug as a duplicate of that bug. 

The present patch does introduce another option (--progress-interval) which is not present in the patches for Bug 2060, so that might be implemented on top of 2060 if someone wants to.

*** This bug has been marked as a duplicate of bug 2060 ***