Bug 11054

Summary: Creating a child record has problems with accented Latin characters
Product: Koha Reporter: Colin Campbell <colin.campbell>
Component: CatalogingAssignee: Colin Campbell <colin.campbell>
Status: CLOSED FIXED QA Contact: Katrin Fischer <katrin.fischer>
Severity: normal    
Priority: P5 - low CC: bgkriegel, giuseppe.angilella, gmcharlt, m.de.rooy, paul.poulain, tomascohen
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:
Attachments: Proposed Patch
[SIGNED-OFF] Bug 11054: Specify utf8 encoding when creating a child record
Amended patch
[SIGNED-OFF] Bug 11054: Specify utf8 encoding when creating a child record
[PASSED QA] Bug 11054: Specify utf8 encoding when creating a child record

Description Colin Campbell 2013-10-15 13:55:48 UTC
When you select New child record from the catalogue detail page. If the data that will be used to create the host field contains latin characters with diacritics from the range covered by latin-1 they are incorrectly encoded in the resulting child record. This is not the case with non-latin characters (Cyrillic, Arabic and Chinese tested) and I suspect with latin diacritic combinations that are not in the latin-1 range
Comment 1 Colin Campbell 2013-10-15 14:32:15 UTC Comment hidden (obsolete)
Comment 2 Bernardo Gonzalez Kriegel 2013-10-15 14:53:58 UTC Comment hidden (obsolete)
Comment 3 Paul Poulain 2013-10-21 21:09:40 UTC
QA comment
 * one-liner
 * I'm wondering if it would not be better to use C4/Charset.pm   sub SetMarcUnicodeFlag($marc_record, $marc_flavour);

for this : it raises the utf8 flag, but ALSO set UNIMARC field 100 if applicable or position 9 of the leader for MARC21/NORMARC

Setting to "in discussion", colin, argue and/or reset status to signed-off
Comment 4 Colin Campbell 2013-10-22 18:24:40 UTC
Paul SetMarcUnicodeFlag sets the MARC flag in the leader the bug is not related to this  the patch is setting the utf-8 flag in the perl var containing the string. A different thing 
Setting back to signed off
Comment 5 Paul Poulain 2013-10-22 18:34:31 UTC
(In reply to Colin Campbell from comment #4)
> Paul SetMarcUnicodeFlag sets the MARC flag in the leader the bug is not
> related to this  the patch is setting the utf-8 flag in the perl var
> containing the string. A different thing 
> Setting back to signed off

Right, but it ALSO set the flag:
    $marc_record->encoding('UTF-8');
(3 lines after the sub prototype)

I still think we should use this sub instead of a direct ->encoding
Comment 6 Bernardo Gonzalez Kriegel 2013-10-22 19:30:01 UTC
For what is worth, with patch applied,
I replaced line 824 in addbiblio.pl

    $record->encoding( 'UTF-8' );

with

    SetMarcUnicodeFlag($record, $marcflavour);

Same (good) result, and perhaps is more general.
Comment 7 Colin Campbell 2013-10-23 14:52:36 UTC Comment hidden (obsolete)
Comment 8 Colin Campbell 2013-10-23 14:54:07 UTC
Switched back to needs sign off. Bernardo it should pass your test
Comment 9 Bernardo Gonzalez Kriegel 2013-10-23 15:01:33 UTC Comment hidden (obsolete)
Comment 10 Katrin Fischer 2013-10-27 15:03:27 UTC
Created attachment 22449 [details] [review]
[PASSED QA] Bug 11054: Specify utf8 encoding when creating a child record

When the only non-ascii characters in a new child record
are latin + diacritic in the range covered by Latin-1 they
are not being added to a new child record correctly encoded.
Explicitly setting the record's encodiing ensures that
they are interpreted correctly.

see "Unicode Bug" in perldoc perlunicode for background on
why these characters are special

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works well. No koha-qa errors.

Same test as before
1) with a record with diacritics in author/title,
2) press new child record, on new record 773adt
with mangled accents
3) Apply patch
4) Repeat 2, now good results

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes all tests and QA script.
Tested with a record containing umlauts in the title,
$t was empty before and umlauts in $a mangled.
After applying the patch fields are correctly filled and
diacritics display correctly.
Comment 11 Galen Charlton 2013-10-31 23:23:47 UTC
Pushed to master.  Thanks, Colin!
Comment 12 Tomás Cohen Arazi (tcohen) 2013-11-20 16:44:51 UTC
This patch has been pushed to 3.12.x, will be in 3.12.7.

Thanks Colin!