Summary: | Koha doesn't check for warnings when parsing (ISO2709) MARC | ||
---|---|---|---|
Product: | Koha | Reporter: | David Cook <dcook> |
Component: | MARC Bibliographic record staging/import | Assignee: | Bugs List <koha-bugs> |
Status: | NEW --- | QA Contact: | Testopia <testopia> |
Severity: | normal | ||
Priority: | P5 - low | ||
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
See Also: |
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=17842 https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=38416 |
||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | --- | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: | Circulation function: |
Description
David Cook
2020-05-25 23:56:02 UTC
Recently, I converted some English/Chinese records in ISO2709 MARC from the Chinese GB2312 encoding to UTF-8 and imported them into Koha. At first, it seemed fine. There were no visible warnings or errors. However, when I looked at the records, it was obvious that many of the Chinese strings were truncated. I figured out that the problem was that MARC::File::USMARC reads the ISO2709 format precisely, which was a problem as I was converting from a 2-byte encoding to a 3-byte encoding. The ISO2709 MARC record label and directory were never going to be correct in that scenario. But Koha didn't complain. So I used MARC::File::USMARC on my own and looked at $record->warnings(), and I saw an explosion of problems with the parsing of my ISO2709 MARC records. In the end, I did a workaround by converting from GB2312 to UTF8 using iconv, opened the ISO2709 MARC in MarcEdit which converted it to its own "MarcBreaker" MRK format, and then exported that as MARCXML. Evidently MarcEdit's MarcBreaker is more skeptical of the ISO2709 MARC format and parses the data more carefully. I am tempted to submit a patch to MARC::File::USMARC to have an option for more cautious parsing, but in the meantime I think we should at least raise the parsing warnings in Koha? Likewise on Bug 17842, I think Katrin was importing a Latin-1 encoded ISO2709 MARC file and Koha didn't complain. It just truncated fields, broke diacritics, etc. In C4::Charset::MarcToUTF8Record, we look for fatal errors, but not non-fatal warnings, which can often be just as bad if not worse... |