Bug 41927

Summary: bulkmarcimport stops if encounters bad record
Product: Koha Reporter: Bernard <bernard.scaife>
Component: Command-line UtilitiesAssignee: Martin Renvoize (ashimema) <martin.renvoize>
Status: RESOLVED DUPLICATE QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: martin.renvoize, robin
Version: 24.11   
Hardware: All   
OS: All   
GIT URL: Initiative type: ---
Sponsorship status: --- Comma delimited list of Sponsors:
Crowdfunding goal: 0 Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Bug Depends on: 35104    
Bug Blocks:    
Attachments: Single badly encoded record example

Description Bernard 2026-02-25 09:52:28 UTC
Created attachment 193837 [details]
Single badly encoded record example

If I have a large file (300k records) to load via bulkmarcimport, it invariably contains a few bad records. The --skip_bad_records does not capture all these and "skip" them. Instead, with some, it reports an error and either stops prematurely or hangs in a loop without being able to progress beyond the offending record until the script is killed. Note that it runs ok in test mode (and identifies the "Bad" record). 

My only recourse is to extract the offending record(s) from the file beforehand which is very time consuming.

What I'd prefer in an ideal world is that these records are sanitized, having captured the error in the log file for later rectification if required. 

As a bare minimum, I need the script to at least be able to handle this as a "skip" as advertised with --skip_bad_records so that it carries on with the rest of the load.

I attach an example of one of the bad records that tripped me up with this:

 /usr/share/koha/bin/migration_tools/bulkmarcimport.pl -commit 1000 -file /kohadevbox/koha/rogue.mrc -l -v -v -sk -si
Characteristic MARC flavour: MARC21
no mapping found for [0x8] at position 11 in t $xtac ar\ $btxt V $2rdac ontent g0=ASCII_DEFAULT g1=EXTENDED_LATIN at /usr/share/perl5/MARC/Charset.pm line 308, <GEN3> line 1.
:36: parser error : PCDATA invalid Char value 8
    <subfield code="a">t $xtac ar\ $btxt V $2rdac ontent</subfield>
                                  ^ at /kohadevbox/koha/Koha/Biblio/Metadata.pm line 117, <GEN3> line 1.
Invalid data, cannot decode metadata object (biblio_metadata.id=453, biblionumber=509, format=marcxml, schema=MARC21, decoding_error=':36: parser error : PCDATA invalid Char value 8
    <subfield code="a">t $xtac ar\ $btxt V $2rdac ontent</subfield>
                                  ^')kohadev-koha@kohadevbox:koha(main)$
Comment 1 Martin Renvoize (ashimema) 2026-02-26 23:50:00 UTC

*** This bug has been marked as a duplicate of bug 35104 ***