Uploading authorities in MARCXML in 16.11 and master kills Umlauts while they work ok in the same file in mrc format. I have not checked biblios yet. Upload mrc: Universität Berlin (Freie Universität) Upload xml: Universitt̃ Berlin (Freie Universitt̃)
Created attachment 58096 [details] Minimal example mrc file
Created attachment 58097 [details] Minimal example xml file
Mirko, I recreate the issue with your file, but I don't if I create an authority record with Koha (110$a Universität Berlin) and I upload them. Where does come from this record?
auth-min.xml: XML 1.0 document, UTF-8 Unicode text from_koha.marcxml: XML 1.0 document, ASCII text
Just had it again. The XML file is created with Catmandu. When I convert it to mrc (with Catmandu) it works.
I can reproduce your problem too (with minimal authority.xml) The solution must be found in RecordsFromMARCXMLFile. Somehow the record format and binary encoding should be passed correctly to MARC::File::XML and friends. Currently, I do not see setting the record format. If it defaults to USMARC, this might already create an issue. And passing the encoding to $batch->next might not be correct as is done now. I tried some things, but they all failed ;)
ping Galen: How should we set record format and binary encoding correctly here? Can't make it up from the cpan documentation. We cannot set the encoding already when we 'use' the module; we should do it later.
Actually the interface mixes format and encoding too. MARC-8 is not an encoding as the interface suggests !
Why not? https://en.wikipedia.org/wiki/MARC-8
(In reply to Katrin Fischer from comment #9) > Why not? > https://en.wikipedia.org/wiki/MARC-8 Well at least confusing :)
Is this still an issue? I love character encoding issues, so I'm happy to examine people's data and try to reproduce (and fix) the problems.
I just have run into this again. We have a MARCXML file that appears to be proper MARC. My editor tells me the encoding is UTF-8 and the umlauts show correctly. On import, the umlauts get broken. Also: we are strangely missing a 245$b where the first word contains an umlaut. If I convert it to .mrc with MARCEdit, which works without any issues or complaints, the file imports correctly.
(In reply to Katrin Fischer from comment #12) > I just have run into this again. We have a MARCXML file that appears to be > proper MARC. My editor tells me the encoding is UTF-8 and the umlauts show > correctly. > > On import, the umlauts get broken. Also: we are strangely missing a 245$b > where the first word contains an umlaut. > > If I convert it to .mrc with MARCEdit, which works without any issues or > complaints, the file imports correctly. I had an issue like that once with a file that was using an older Chinese encoding. I tried converting it to UTF-8 with all kinds of tools and nothing produced made Koha happy. However, when I did the conversion in MARCEdit, I was able to import it into Koha with no problem. This was a while ago so I don't remember what the finer points were...
I think I solved the riddle: The problem appears, if LDR pos. 9 = a (encoding) is missing. The attached MARCXML record here is missing the leader and this position was unset in the records I had issues with. Once I fixed the records, the import worked as expected. I wonder if we could automatically assume UTF-8 encoding, if the encoding is not set?
I was able to reproduce in current master (22.12.00.013). I tried Katrin's trick of editing manually the record in vim and it imported correctly. However, I only had one record in my file... Not sure it's reasonable to edit manually files with lots of records, especially the leader, it can be finnicky. I'm not even sure the leader I changed it to is correct... I will attach the record. Here's what I did: 1. Download attached file 2. Optionally, open the file in a text editor (I used vim) --> The encoding is specified in the first line <?xml version="1.0" encoding="UTF-8"?> --> Accents are displayed correctly 3. In the staff interface, go to Cataloging > Stage MARC records for import 4. Choose the file 5. Click Upload file 6. In the form, choose Format: MARCXML (leave the rest as is) 7. Click Stage for import 8. Click View batch --> In the table, accents are displayed correctly 9. Click on the title --> Accents are *not* displayed correctly 10. Click Import this batch into the catalog 11. Click Manage imported batch 12. Click View in the last column of the table --> Accents are *not* displayed correctly in the record details --> Accents are displayed correctly in the breadcrumbs, page/tab title 13. Click the MARC tab --> Accents are displayed correctly in the page header --> Accents are *not* displayed correctly in the record details (tab 2 for example) 14. Edit the file in a text editor (again I used vim) and replace the leader by 28152nam a2201417 i 4500 15. Try steps 3 to 13 again --> Accents are ok everywhere
Created attachment 148663 [details] MARCXML file with one record Attached record for my test plan is previous comment.
(In reply to Katrin Fischer from comment #14) > I wonder if we could automatically assume UTF-8 encoding, if the encoding is > not set? We specify the encoding in the form when we stage the file. Should this override the encoding in the records?
Looking at the source code for the Perl MARC::File::XML module that is used to read the XML file and build MARC records, the ninth character of the leader *must* be an 'a'. Otherwise, even if we pass the correct encoding to the MARC::Record::new_from_xml function, if the ninth character is not an 'a', the result will not be processed as UTF-8. I am no expert in MARCXML, but according to the schema (https://www.loc.gov/standards/marcxml/schema/MARC21slim.xsd), each character of the leader must be present. I think that's why the MARC::File::XML module assumes that there will be either a space or an 'a' character at the ninth position. The possible solutions to this issue, as I see them, would be to: a) reject a file with an incorrect leader. b) modify the leader before importing to correct any existing mistake, although I don't see how we would be able to determine the missing character(s). c) find another module to import MARC records. d) become part of the team working on MARC::File::XML and change the source code :)
Should we close this as wontfix?