Bug 38933

Summary: Some oversized records with UTF-8 characters cause import worker to die
Product: Koha Reporter: Janusz Kaczmarek <januszop>
Component: MARC Bibliographic record staging/importAssignee: Bugs List <koha-bugs>
Status: NEW --- QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: dcook, martin.renvoize
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38913
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=35104
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Test record

Description Janusz Kaczmarek 2025-01-20 22:16:12 UTC
Some oversized records with UTF-8 characters cause import worker to die.  The record would be staged correctly, but then an attempt to import it with C4::ImportBatch::BatchCommitRecords can generate an uncaught die of new_from_usmarc.

A solution would be to make use of import_records.marcxml instead of import_records.marc for this step. (Both versions are stored in the import_records table in stage step).
Comment 1 Janusz Kaczmarek 2025-01-20 22:19:11 UTC
Created attachment 176845 [details]
Test record

A test MARCXML record to confirm the issue.
Comment 2 David Cook 2025-01-20 22:41:02 UTC
I was actually thinking about this a bit yesterday when fixing bug 38913.

There are other places that call "new_from_usmarc". Some of them - like the API - have try/catch around them I think, but I'm sure not all the calls do.
Comment 3 Janusz Kaczmarek 2025-01-20 22:52:24 UTC
(In reply to David Cook from comment #2)
> I was actually thinking about this a bit yesterday when fixing bug 38913.
> 
> There are other places that call "new_from_usmarc". Some of them - like the
> API - have try/catch around them I think, but I'm sure not all the calls do.

Should be investigated from this angle.  But the current can be solved with marcxml that we have anyway.  No need to try here around new_from_usmarc if we can effectively import such records with new_from_xml IMO.