Bug 41927 - bulkmarcimport stops if encounters bad record
Summary: bulkmarcimport stops if encounters bad record
Status: RESOLVED DUPLICATE of bug 35104
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: 24.11
Hardware: All All
: P5 - low normal
Assignee: Martin Renvoize (ashimema)
QA Contact: Testopia
URL:
Keywords:
Depends on: 35104
Blocks:
  Show dependency treegraph
 
Reported: 2026-02-25 09:52 UTC by Bernard
Modified: 2026-02-26 23:50 UTC (History)
2 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Single badly encoded record example (1.75 KB, application/marc)
2026-02-25 09:52 UTC, Bernard
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Bernard 2026-02-25 09:52:28 UTC
Created attachment 193837 [details]
Single badly encoded record example

If I have a large file (300k records) to load via bulkmarcimport, it invariably contains a few bad records. The --skip_bad_records does not capture all these and "skip" them. Instead, with some, it reports an error and either stops prematurely or hangs in a loop without being able to progress beyond the offending record until the script is killed. Note that it runs ok in test mode (and identifies the "Bad" record). 

My only recourse is to extract the offending record(s) from the file beforehand which is very time consuming.

What I'd prefer in an ideal world is that these records are sanitized, having captured the error in the log file for later rectification if required. 

As a bare minimum, I need the script to at least be able to handle this as a "skip" as advertised with --skip_bad_records so that it carries on with the rest of the load.

I attach an example of one of the bad records that tripped me up with this:

 /usr/share/koha/bin/migration_tools/bulkmarcimport.pl -commit 1000 -file /kohadevbox/koha/rogue.mrc -l -v -v -sk -si
Characteristic MARC flavour: MARC21
no mapping found for [0x8] at position 11 in t $xtac ar\ $btxt V $2rdac ontent g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308, <GEN3> line 1.
:36: parser error : PCDATA invalid Char value 8
    <subfield code="a">t $xtac ar\ $btxt V $2rdac ontent</subfield>
                                  ^ at /kohadevbox/koha/Koha/Biblio/Metadata.pm line 117, <GEN3> line 1.
Invalid data, cannot decode metadata object (biblio_metadata.id=453, biblionumber=509, format=marcxml, schema=MARC21, decoding_error=':36: parser error : PCDATA invalid Char value 8
    <subfield code="a">t $xtac ar\ $btxt V $2rdac ontent</subfield>
                                  ^')kohadev-koha@kohadevbox:koha(main)$
Comment 1 Martin Renvoize (ashimema) 2026-02-26 23:50:00 UTC

*** This bug has been marked as a duplicate of bug 35104 ***