Bug 25600 - Koha doesn't check for warnings when parsing (ISO2709) MARC
Summary: Koha doesn't check for warnings when parsing (ISO2709) MARC
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic record staging/import (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-05-25 23:56 UTC by David Cook
Modified: 2020-05-26 00:14 UTC (History)
0 users

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2020-05-25 23:56:02 UTC
I've noticed that Koha doesn't check for warnings when parsing MARC, particularly ISO2709 MARC, which is a more fragile format than MARCXML.

We should be using https://metacpan.org/pod/MARC::Record#warnings() to at the very least warn users that there was an issue with their (ISO2709) MARC import.
Comment 1 David Cook 2020-05-26 00:02:47 UTC
Recently, I converted some English/Chinese records in ISO2709 MARC from the Chinese GB2312 encoding to UTF-8 and imported them into Koha. 

At first, it seemed fine. There were no visible warnings or errors. 

However, when I looked at the records, it was obvious that many of the Chinese strings were truncated.

I figured out that the problem was that MARC::File::USMARC reads the ISO2709 format precisely, which was a problem as I was converting from a 2-byte encoding to a 3-byte encoding. The ISO2709 MARC record label and directory were never going to be correct in that scenario.

But Koha didn't complain.

So I used MARC::File::USMARC on my own and looked at $record->warnings(), and I saw an explosion of problems with the parsing of my ISO2709 MARC records.

In the end, I did a workaround by converting from GB2312 to UTF8 using iconv, opened the ISO2709 MARC in MarcEdit which converted it to its own "MarcBreaker" MRK format, and then exported that as MARCXML. Evidently MarcEdit's MarcBreaker is more skeptical of the ISO2709 MARC format and parses the data more carefully. 

I am tempted to submit a patch to MARC::File::USMARC to have an option for more cautious parsing, but in the meantime I think we should at least raise the parsing warnings in Koha?
Comment 2 David Cook 2020-05-26 00:04:21 UTC
Likewise on Bug 17842, I think Katrin was importing a Latin-1 encoded ISO2709 MARC file and Koha didn't complain. It just truncated fields, broke diacritics, etc.
Comment 3 David Cook 2020-05-26 00:14:18 UTC
In C4::Charset::MarcToUTF8Record, we look for fatal errors, but not non-fatal warnings, which can often be just as bad if not worse...