Bug 17754 - MARCXML upload in Intranet is broken (encoding) if LDR pos. 9 is missing
Summary: MARCXML upload in Intranet is broken (encoding) if LDR pos. 9 is missing
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Tools (show other bugs)
Version: 20.11
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-12-10 12:04 UTC by Mirko Tietgen
Modified: 2023-03-29 18:34 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Minimal example mrc file (179 bytes, text/plain)
2016-12-10 12:05 UTC, Mirko Tietgen
Details
Minimal example xml file (545 bytes, text/xml)
2016-12-10 12:05 UTC, Mirko Tietgen
Details
MARCXML file with one record (6.28 KB, text/xml)
2023-03-24 14:20 UTC, Caroline Cyr La Rose
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Mirko Tietgen 2016-12-10 12:04:38 UTC
Uploading authorities in MARCXML in 16.11 and master kills Umlauts while they work ok in the same file in mrc format. I have not checked biblios yet.

Upload mrc: Universität Berlin (Freie Universität)
Upload xml: Universitt̃ Berlin (Freie Universitt̃)
Comment 1 Mirko Tietgen 2016-12-10 12:05:12 UTC
Created attachment 58096 [details]
Minimal example mrc file
Comment 2 Mirko Tietgen 2016-12-10 12:05:39 UTC
Created attachment 58097 [details]
Minimal example xml file
Comment 3 Jonathan Druart 2016-12-20 15:33:50 UTC
Mirko, I recreate the issue with your file, but I don't if I create an authority record with Koha (110$a Universität Berlin) and I upload them.
Where does come from this record?
Comment 4 Jonathan Druart 2016-12-20 15:35:06 UTC
auth-min.xml: XML 1.0 document, UTF-8 Unicode text

from_koha.marcxml: XML 1.0 document, ASCII text
Comment 5 Mirko Tietgen 2017-02-02 19:25:10 UTC
Just had it again. The XML file is created with Catmandu. When I convert it to mrc (with Catmandu) it works.
Comment 6 Marcel de Rooy 2017-02-03 08:05:58 UTC
I can reproduce your problem too (with minimal authority.xml)

The solution must be found in RecordsFromMARCXMLFile.
Somehow the record format and binary encoding should be passed correctly to MARC::File::XML and friends.
Currently, I do not see setting the record format. If it defaults to USMARC, this might already create an issue.
And passing the encoding to $batch->next might not be correct as is done now.
I tried some things, but they all failed ;)
Comment 7 Marcel de Rooy 2017-02-03 08:09:53 UTC
ping Galen: How should we set record format and binary encoding correctly here? Can't make it up from the cpan documentation. We cannot set the encoding already when we 'use' the module; we should do it later.
Comment 8 Marcel de Rooy 2017-02-03 08:11:51 UTC
Actually the interface mixes format and encoding too. MARC-8 is not an encoding as the interface suggests !
Comment 9 Katrin Fischer 2017-02-05 16:18:21 UTC
Why not?
https://en.wikipedia.org/wiki/MARC-8
Comment 10 Marcel de Rooy 2017-02-06 07:26:52 UTC
(In reply to Katrin Fischer from comment #9)
> Why not?
> https://en.wikipedia.org/wiki/MARC-8

Well at least confusing :)
Comment 11 David Cook 2020-05-26 00:06:47 UTC
Is this still an issue?

I love character encoding issues, so I'm happy to examine people's data and try to reproduce (and fix) the problems.
Comment 12 Katrin Fischer 2022-06-07 13:12:16 UTC
I just have run into this again. We have a MARCXML file that appears to be proper MARC. My editor tells me the encoding is UTF-8 and the umlauts show correctly.

On import, the umlauts get broken. Also: we are strangely missing a 245$b where the first word contains an umlaut.

If I convert it to .mrc with MARCEdit, which works without any issues or complaints, the file imports correctly.
Comment 13 David Cook 2022-06-27 01:43:27 UTC
(In reply to Katrin Fischer from comment #12)
> I just have run into this again. We have a MARCXML file that appears to be
> proper MARC. My editor tells me the encoding is UTF-8 and the umlauts show
> correctly.
> 
> On import, the umlauts get broken. Also: we are strangely missing a 245$b
> where the first word contains an umlaut.
> 
> If I convert it to .mrc with MARCEdit, which works without any issues or
> complaints, the file imports correctly.

I had an issue like that once with a file that was using an older Chinese encoding. I tried converting it to UTF-8 with all kinds of tools and nothing produced made Koha happy. 

However, when I did the conversion in MARCEdit, I was able to import it into Koha with no problem. 

This was a while ago so I don't remember what the finer points were...
Comment 14 Katrin Fischer 2022-07-15 15:55:14 UTC
I think I solved the riddle: The problem appears, if LDR pos. 9 = a (encoding) is missing. 

The attached MARCXML record here is missing the leader and this position was unset in the records I had issues with. Once I fixed the records, the import worked as expected.

I wonder if we could automatically assume UTF-8 encoding, if the encoding is not set?
Comment 15 Caroline Cyr La Rose 2023-03-24 14:19:06 UTC
I was able to reproduce in current master (22.12.00.013). I tried Katrin's trick of editing manually the record in vim and it imported correctly. However, I only had one record in my file... Not sure it's reasonable to edit manually files with lots of records, especially the leader, it can be finnicky. I'm not even sure the leader I changed it to is correct...

I will attach the record.

Here's what I did:
1. Download attached file
2. Optionally, open the file in a text editor (I used vim)
   --> The encoding is specified in the first line <?xml version="1.0" encoding="UTF-8"?>
   --> Accents are displayed correctly
3. In the staff interface, go to Cataloging > Stage MARC records for import
4. Choose the file
5. Click Upload file
6. In the form, choose Format: MARCXML (leave the rest as is)
7. Click Stage for import
8. Click View batch
   --> In the table, accents are displayed correctly
9. Click on the title
   --> Accents are *not* displayed correctly
10. Click Import this batch into the catalog
11. Click Manage imported batch
12. Click View in the last column of the table
   --> Accents are *not* displayed correctly in the record details
   --> Accents are displayed correctly in the breadcrumbs, page/tab title
13. Click the MARC tab
   --> Accents are displayed correctly in the page header
   --> Accents are *not* displayed correctly in the record details (tab 2 for example)
14. Edit the file in a text editor (again I used vim) and replace the leader by
28152nam a2201417 i 4500
15. Try steps 3 to 13 again
   --> Accents are ok everywhere
Comment 16 Caroline Cyr La Rose 2023-03-24 14:20:08 UTC
Created attachment 148663 [details]
MARCXML file with one record

Attached record for my test plan is previous comment.
Comment 17 Caroline Cyr La Rose 2023-03-24 14:28:36 UTC
(In reply to Katrin Fischer from comment #14)

> I wonder if we could automatically assume UTF-8 encoding, if the encoding is
> not set?

We specify the encoding in the form when we stage the file. Should this override the encoding in the records?
Comment 18 Olivier Hubert 2023-03-29 18:18:58 UTC
Looking at the source code for the Perl MARC::File::XML module that is used to read the XML file and build MARC records, the ninth character of the leader *must* be an 'a'. Otherwise, even if we pass the correct encoding to the MARC::Record::new_from_xml function, if the ninth character is not an 'a', the result will not be processed as UTF-8.

I am no expert in MARCXML, but according to the schema (https://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd), each character of the leader must be present. I think that's why the MARC::File::XML module assumes that there will be either a space or an 'a' character at the ninth position.

The possible solutions to this issue, as I see them, would be to:
a) reject a file with an incorrect leader.
b) modify the leader before importing to correct any existing mistake, although I don't see how we would be able to determine the missing character(s).
c) find another module to import MARC records.
d) become part of the team working on MARC::File::XML and change the source code :)
Comment 19 Caroline Cyr La Rose 2023-03-29 18:34:19 UTC
Should we close this as wontfix?