Bug 38933 - Some oversized records with UTF-8 characters cause import worker to die
Summary: Some oversized records with UTF-8 characters cause import worker to die
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low major
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-01-20 22:16 UTC by Janusz Kaczmarek
Modified: 2025-01-21 16:38 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Test record (77.45 KB, application/x-zip-compressed)
2025-01-20 22:19 UTC, Janusz Kaczmarek
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Janusz Kaczmarek 2025-01-20 22:16:12 UTC
Some oversized records with UTF-8 characters cause import worker to die.  The record would be staged correctly, but then an attempt to import it with C4::ImportBatch::BatchCommitRecords can generate an uncaught die of new_from_usmarc.

A solution would be to make use of import_records.marcxml instead of import_records.marc for this step. (Both versions are stored in the import_records table in stage step).
Comment 1 Janusz Kaczmarek 2025-01-20 22:19:11 UTC
Created attachment 176845 [details]
Test record

A test MARCXML record to confirm the issue.
Comment 2 David Cook 2025-01-20 22:41:02 UTC
I was actually thinking about this a bit yesterday when fixing bug 38913.

There are other places that call "new_from_usmarc". Some of them - like the API - have try/catch around them I think, but I'm sure not all the calls do.
Comment 3 Janusz Kaczmarek 2025-01-20 22:52:24 UTC
(In reply to David Cook from comment #2)
> I was actually thinking about this a bit yesterday when fixing bug 38913.
> 
> There are other places that call "new_from_usmarc". Some of them - like the
> API - have try/catch around them I think, but I'm sure not all the calls do.

Should be investigated from this angle.  But the current can be solved with marcxml that we have anyway.  No need to try here around new_from_usmarc if we can effectively import such records with new_from_xml IMO.