Attachment 173926 Details for Bug 37478 – Bug 37478: Rename strict to skip_bad_records and add POD

[patch] Bug 37478: Rename strict to skip_bad_records and add POD

Bug-37478-Rename-strict-to-skipbadrecords-and-add-.patch (text/plain), 4.68 KB, created by Martin Renvoize (ashimema) on 2024-11-04 14:49:31 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Martin Renvoize (ashimema)

Created: 2024-11-04 14:49:31 UTC

Size: 4.68 KB

patch

obsolete

>From 469b84c98616215fc6500f0a49797a47bf0a567b Mon Sep 17 00:00:00 2001
>From: Nick Clemens <nick@bywatersolutions.com>
>Date: Thu, 31 Oct 2024 19:23:08 +0000
>Subject: [PATCH] Bug 37478: Rename strict to skip_bad_records and add POD
>
>Option name changed, POD updated.
>The reason not to do this by default is that it is an extra record conversion
>to check the validity, so may slow down the import.
>
>Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
>---
> misc/migration_tools/bulkmarcimport.pl | 80 ++++++++++++++------------
> 1 file changed, 43 insertions(+), 37 deletions(-)
>
>diff --git a/misc/migration_tools/bulkmarcimport.pl b/misc/migration_tools/bulkmarcimport.pl
>index 691e9c9e363..4406252fc7d 100755
>--- a/misc/migration_tools/bulkmarcimport.pl
>+++ b/misc/migration_tools/bulkmarcimport.pl
>@@ -77,45 +77,45 @@ my $localcust;
> my $marc_mod_template    = '';
> my $marc_mod_template_id = -1;
> my $skip_indexing        = 0;
>-my $strict_mode;
>+my $skip_bad_records;
> $| = 1;
> 
> GetOptions(
>-    'commit:f'          => \$commit,
>-    'file:s'            => \$input_marc_file,
>-    'n:f'               => \$number,
>-    'o|offset:f'        => \$offset,
>-    'h|help'            => \$version,
>-    'd|delete'          => \$delete,
>-    't|test'            => \$test_parameter,
>-    's'                 => \$skip_marc8_conversion,
>-    'c:s'               => \$char_encoding,
>-    'v|verbose:+'       => \$verbose,
>-    'fk'                => \$fk_off,
>-    'm:s'               => \$format,
>-    'l:s'               => \$logfile,
>-    'append'            => \$append,
>-    'k|keepids:s'       => \$keepids,
>-    'b|biblios'         => \$biblios,
>-    'a|authorities'     => \$authorities,
>-    'authtypes:s'       => \$authtypes,
>-    'filter=s@'         => \$filters,
>-    'insert'            => \$insert,
>-    'update'            => \$update,
>-    'all'               => \$all,
>-    'match=s@'          => \$match,
>-    'i|isbn'            => \$isbn_check,
>-    'x:s'               => \$sourcetag,
>-    'y:s'               => \$sourcesubfield,
>-    'idmap:s'           => \$idmapfl,
>-    'cleanisbn!'        => \$cleanisbn,
>-    'yaml:s'            => \$yamlfile,
>-    'dedupbarcode'      => \$dedup_barcode,
>-    'framework=s'       => \$framework,
>-    'custom:s'          => \$localcust,
>-    'marcmodtemplate:s' => \$marc_mod_template,
>-    'si|skip_indexing'  => \$skip_indexing,
>-    'st|strict'         => \$strict_mode,
>+    'commit:f'            => \$commit,
>+    'file:s'              => \$input_marc_file,
>+    'n:f'                 => \$number,
>+    'o|offset:f'          => \$offset,
>+    'h|help'              => \$version,
>+    'd|delete'            => \$delete,
>+    't|test'              => \$test_parameter,
>+    's'                   => \$skip_marc8_conversion,
>+    'c:s'                 => \$char_encoding,
>+    'v|verbose:+'         => \$verbose,
>+    'fk'                  => \$fk_off,
>+    'm:s'                 => \$format,
>+    'l:s'                 => \$logfile,
>+    'append'              => \$append,
>+    'k|keepids:s'         => \$keepids,
>+    'b|biblios'           => \$biblios,
>+    'a|authorities'       => \$authorities,
>+    'authtypes:s'         => \$authtypes,
>+    'filter=s@'           => \$filters,
>+    'insert'              => \$insert,
>+    'update'              => \$update,
>+    'all'                 => \$all,
>+    'match=s@'            => \$match,
>+    'i|isbn'              => \$isbn_check,
>+    'x:s'                 => \$sourcetag,
>+    'y:s'                 => \$sourcesubfield,
>+    'idmap:s'             => \$idmapfl,
>+    'cleanisbn!'          => \$cleanisbn,
>+    'yaml:s'              => \$yamlfile,
>+    'dedupbarcode'        => \$dedup_barcode,
>+    'framework=s'         => \$framework,
>+    'custom:s'            => \$localcust,
>+    'marcmodtemplate:s'   => \$marc_mod_template,
>+    'si|skip_indexing'    => \$skip_indexing,
>+    'sk|skip_bad_records' => \$skip_bad_records,
> );
> 
> $biblios ||= !$authorities;
>@@ -343,7 +343,7 @@ RECORD: while () {
>     }
>     if ($record) {
> 
>-        if ($strict_mode) {
>+        if ($skip_bad_records) {
>             my $xml = $record->as_xml_record();
>             eval { MARC::Record::new_from_xml( $xml, 'UTF-8', "MARC21" ); };
>             if ($@) {
>@@ -1044,6 +1044,12 @@ If set, do not index the imported records with Zebra or Elasticsearch.
> Use this when you plan to do a complete reindex of your data after running
> bulkmarciport. This can increase performance and avoid unnecessary load.
> 
>+=item B<-sk, --skip_bad_records>
>+
>+If set, check the validity of records before adding. If they are invalid we will
>+print the outout of MARC::Lint->check_record and skip them during the import. Without
>+this option bad records may kill the job.
>+
> =back
> 
> =cut
>-- 
>2.47.0

Actions: View | Diff | Splinter Review

Attachments on bug 37478: 169598 | 169599 | 169600 | 170601 | 170703 | 170704 | 173831 | 173832 | 173833 | 173924 | 173925 | 173926