Created attachment 50819 [details] [review] TransformHtmlToMarc-UTF8.patch Hello, Using Koha 3.22.05 (Debian 8 packaging), I noticed that creating a new authority record in the web interface would lead to character encoding inconsistencies when the leader is not included in the MARC editor (which I felt is not necessary for authority records): in the database, the "auth_header.marc" (blob) would still have a "MARC-8" leader, despite the content being correctly UTF-8 encoded. I tracked the issue down to the new MARC record not being initialized as UTF-8 encoded when being created from the web interface (=> TransformHtmlToMarc). The one-liner attached patch solved that issue. Best, Cédric
Hi Cédric, Please have a look at the following wiki patch to know how to correctly format a patch: https://wiki.koha-community.org/wiki/SubmitingAPatch It would be also good to provide a test plan to reproduce the issue.
I can tell you how to reproduce this, it happens in the standard MARC editor as well as the authority editor. Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework. create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing. Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it. If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not. As Cédric says, it seems to be because the leader is set to MARC-8 in the absence of the leader, instead of UTF8.
Created attachment 53254 [details] Framework for testing, no 000 visible
Created attachment 53255 [details] [review] Bug 16370 - Frameworks without 000 visible won't save UTF8 characters Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework. Create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing. Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it. Apply the patch, redo above, fields with UTF8 Characters will be saved correctly Note (from Liz): I have no idea if this is the right solution, but it does fix the problem, so I'm formatting this as required for testing/inclusion on behalf of Cédric. If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not.
Created attachment 53256 [details] [review] Bug 16370 - Frameworks without 000 visible won't save UTF8 characters Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework. Create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing. Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it, and no longer appears in the editor. If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not. Apply the patch, redo above, fields with UTF8 Characters will be saved correctly Note (from Liz): I have no idea if this is the right solution, but it does fix the problem, so I'm formatting this as required for testing/inclusion on behalf of Cédric.
Comment on attachment 50819 [details] [review] TransformHtmlToMarc-UTF8.patch Liz Rea's patch is identical and git-friendlier for testing.
I even kept the attribution! :)
> Create a framework that doesn't include the 000 in the editor (set to ignore) > alternatively, import the attached framework. Hmm. Isn't this kind of a strange use case? I have doubts on correcting the problem in TransformHtmlToMarc. Currently, the marc21 leader plugin forces the leader position to Unicode. But what if you edit the leader manually? (Just as strange as ignoring the leader.) And what are the implications for UNIMARC? Note that the encoding for that is not in the leader but in field 100.