Bug 16370

Summary: Set the record encoding to UTF8 when TransformHtmlToMarc
Product: Koha Reporter: Cédric Dufour <cdufour-keyword-koha.b708e6>
Component: MARC Bibliographic data supportAssignee: Galen Charlton <gmcharlt>
Status: Failed QA --- QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: jonathan.druart, liz, m.de.rooy
Version: 3.22   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: TransformHtmlToMarc-UTF8.patch
Framework for testing, no 000 visible
Bug 16370 - Frameworks without 000 visible won't save UTF8 characters
Bug 16370 - Frameworks without 000 visible won't save UTF8 characters

Description Cédric Dufour 2016-04-27 15:24:36 UTC Comment hidden (obsolete)
Comment 1 Jonathan Druart 2016-05-02 14:59:27 UTC
Hi Cédric,
Please have a look at the following wiki patch to know how to correctly format a patch: https://wiki.koha-community.org/wiki/SubmitingAPatch
It would be also good to provide a test plan to reproduce the issue.
Comment 2 Liz Rea 2016-07-11 01:43:13 UTC
I can tell you how to reproduce this, it happens in the standard MARC editor as well as the authority editor.

Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework.

create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing.

Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it. 

If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not.

As Cédric says, it seems to be because the leader is set to MARC-8 in the absence of the leader, instead of UTF8.
Comment 3 Liz Rea 2016-07-11 01:43:42 UTC
Created attachment 53254 [details]
Framework for testing, no 000 visible
Comment 4 Liz Rea 2016-07-11 02:27:10 UTC Comment hidden (obsolete)
Comment 5 Liz Rea 2016-07-11 02:29:15 UTC
Created attachment 53256 [details] [review]
Bug 16370 - Frameworks without 000 visible won't save UTF8 characters

Create a framework that doesn't include the 000 in the editor (set to ignore) alternatively, import the attached framework.

Create or import a record that has UTF8 characters in it into that framework. I used āāāāāāā for the title, because that's easy to spot disappearing.

Edit the record, and modify any field. You'll notice that the title is no longer displayed after you save it, and no longer appears in the editor.

If you look in the database in biblioitems for this biblio, you'll spot that the MARC has the title saved, but the MARCXML does not.

Apply the patch, redo above, fields with UTF8 Characters will be saved correctly

Note (from Liz): I have no idea if this is the right solution, but it does fix the problem, so I'm formatting this as required for testing/inclusion on behalf of Cédric.
Comment 6 Mark Tompsett 2016-07-26 03:39:10 UTC
Comment on attachment 50819 [details] [review]
TransformHtmlToMarc-UTF8.patch

Liz Rea's patch is identical and git-friendlier for testing.
Comment 7 Liz Rea 2016-07-26 03:41:22 UTC
I even kept the attribution! :)
Comment 8 Marcel de Rooy 2016-08-18 11:13:49 UTC
> Create a framework that doesn't include the 000 in the editor (set to ignore)
> alternatively, import the attached framework.

Hmm. Isn't this kind of a strange use case?

I have doubts on correcting the problem in TransformHtmlToMarc.
Currently, the marc21 leader plugin forces the leader position to Unicode. But what if you edit the leader manually? (Just as strange as ignoring the leader.)

And what are the implications for UNIMARC? Note that the encoding for that is not in the leader but in field 100.