Bugzilla – Attachment 93826 Details for
Bug 23766
Allow MARC record importer to handle large batches without causing memory issues
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 23766: Upate RecordsFromISO2709File and RecordsFromMARCXMLFile to accept an offset and count
Bug-23766-Upate-RecordsFromISO2709File-and-Records.patch (text/plain), 2.68 KB, created by
Kyle M Hall (khall)
on 2019-10-07 15:54:50 UTC
(
hide
)
Description:
Bug 23766: Upate RecordsFromISO2709File and RecordsFromMARCXMLFile to accept an offset and count
Filename:
MIME Type:
Creator:
Kyle M Hall (khall)
Created:
2019-10-07 15:54:50 UTC
Size:
2.68 KB
patch
obsolete
>From c11f166ed16df80023bd71323e9edec85ec7cb1f Mon Sep 17 00:00:00 2001 >From: Kyle M Hall <kyle@bywatersolutions.com> >Date: Mon, 7 Oct 2019 11:54:31 -0400 >Subject: [PATCH] Bug 23766: Upate RecordsFromISO2709File and > RecordsFromMARCXMLFile to accept an offset and count > >--- > C4/ImportBatch.pm | 22 ++++++++++++++++------ > 1 file changed, 16 insertions(+), 6 deletions(-) > >diff --git a/C4/ImportBatch.pm b/C4/ImportBatch.pm >index 8d39ed2034..6211de0284 100644 >--- a/C4/ImportBatch.pm >+++ b/C4/ImportBatch.pm >@@ -1495,7 +1495,8 @@ Returns two array refs. > =cut > > sub RecordsFromISO2709File { >- my ($input_file, $record_type, $encoding) = @_; >+ my ($input_file, $record_type, $encoding, $offset, $count) = @_; >+ $offset ||= 0; > my @errors; > > my $marc_type = C4::Context->preference('marcflavour'); >@@ -1507,6 +1508,7 @@ sub RecordsFromISO2709File { > while (<IN>) { > s/^\s+//; > s/\s+$//; >+ next unless $i >= $offset; > next unless $_; # skip if record has only whitespace, as might occur > # if file includes newlines between each MARC record > my ($marc_record, $charset_guessed, $char_errors) = MarcToUTF8Record($_, $marc_type, $encoding); >@@ -1515,6 +1517,7 @@ sub RecordsFromISO2709File { > push @errors, > "Unexpected charset $charset_guessed, expecting $encoding"; > } >+ last unless ( !$count && !$offset ) || ( $count && $i < $count + $offset ); > } > close IN; > return ( \@errors, \@marc_records ); >@@ -1522,7 +1525,7 @@ sub RecordsFromISO2709File { > > =head2 RecordsFromMARCXMLFile > >- my ($errors, $records) = C4::ImportBatch::RecordsFromMARCXMLFile($input_file, $encoding); >+ my ($errors, $records) = C4::ImportBatch::RecordsFromMARCXMLFile($input_file, $encoding, $offset, $count); > > Creates MARC::Record-objects out of the given MARCXML-file. > >@@ -1534,16 +1537,23 @@ Returns two array refs. > =cut > > sub RecordsFromMARCXMLFile { >- my ( $filename, $encoding ) = @_; >+ my ( $filename, $encoding, $offset, $count ) = @_; >+ $offset ||= 0; >+ >+ $offset = 100; >+ $count = 100; > my $batch = MARC::File::XML->in( $filename ); >+ > my ( @marcRecords, @errors, $record ); >+ my $i = 0; > do { > eval { $record = $batch->next( $encoding ); }; > if ($@) { >- push @errors, $@; >+ push( @errors, $@ ) if $i >= $offset; > } >- push @marcRecords, $record if $record; >- } while( $record ); >+ push @marcRecords, $record if $record && $i >= $offset; >+ $i++; >+ } while( $record && ( ( !$count && !$offset ) || ( $count && $i < $count + $offset ) ) ); > return (\@errors, \@marcRecords); > } > >-- >2.21.0 (Apple Git-122)
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 23766
: 93826