Bug 11054 - Creating a child record has problems with accented Latin characters
Summary: Creating a child record has problems with accented Latin characters
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Colin Campbell
QA Contact: Katrin Fischer
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-10-15 13:55 UTC by Colin Campbell
Modified: 2015-06-04 23:24 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Proposed Patch (1.15 KB, patch)
2013-10-15 14:32 UTC, Colin Campbell
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 11054: Specify utf8 encoding when creating a child record (1.57 KB, patch)
2013-10-15 14:53 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Amended patch (1.16 KB, patch)
2013-10-23 14:52 UTC, Colin Campbell
Details | Diff | Splinter Review
[SIGNED-OFF] Bug 11054: Specify utf8 encoding when creating a child record (1.45 KB, patch)
2013-10-23 15:01 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
[PASSED QA] Bug 11054: Specify utf8 encoding when creating a child record (1.72 KB, patch)
2013-10-27 15:03 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Colin Campbell 2013-10-15 13:55:48 UTC
When you select New child record from the catalogue detail page. If the data that will be used to create the host field contains latin characters with diacritics from the range covered by latin-1 they are incorrectly encoded in the resulting child record. This is not the case with non-latin characters (Cyrillic, Arabic and Chinese tested) and I suspect with latin diacritic combinations that are not in the latin-1 range
Comment 1 Colin Campbell 2013-10-15 14:32:15 UTC Comment hidden (obsolete)
Comment 2 Bernardo Gonzalez Kriegel 2013-10-15 14:53:58 UTC Comment hidden (obsolete)
Comment 3 Paul Poulain 2013-10-21 21:09:40 UTC
QA comment
 * one-liner
 * I'm wondering if it would not be better to use C4/Charset.pm   sub SetMarcUnicodeFlag($marc_record, $marc_flavour);

for this : it raises the utf8 flag, but ALSO set UNIMARC field 100 if applicable or position 9 of the leader for MARC21/NORMARC

Setting to "in discussion", colin, argue and/or reset status to signed-off
Comment 4 Colin Campbell 2013-10-22 18:24:40 UTC
Paul SetMarcUnicodeFlag sets the MARC flag in the leader the bug is not related to this  the patch is setting the utf-8 flag in the perl var containing the string. A different thing 
Setting back to signed off
Comment 5 Paul Poulain 2013-10-22 18:34:31 UTC
(In reply to Colin Campbell from comment #4)
> Paul SetMarcUnicodeFlag sets the MARC flag in the leader the bug is not
> related to this  the patch is setting the utf-8 flag in the perl var
> containing the string. A different thing 
> Setting back to signed off

Right, but it ALSO set the flag:
    $marc_record->encoding('UTF-8');
(3 lines after the sub prototype)

I still think we should use this sub instead of a direct ->encoding
Comment 6 Bernardo Gonzalez Kriegel 2013-10-22 19:30:01 UTC
For what is worth, with patch applied,
I replaced line 824 in addbiblio.pl

    $record->encoding( 'UTF-8' );

with

    SetMarcUnicodeFlag($record, $marcflavour);

Same (good) result, and perhaps is more general.
Comment 7 Colin Campbell 2013-10-23 14:52:36 UTC Comment hidden (obsolete)
Comment 8 Colin Campbell 2013-10-23 14:54:07 UTC
Switched back to needs sign off. Bernardo it should pass your test
Comment 9 Bernardo Gonzalez Kriegel 2013-10-23 15:01:33 UTC Comment hidden (obsolete)
Comment 10 Katrin Fischer 2013-10-27 15:03:27 UTC
Created attachment 22449 [details] [review]
[PASSED QA] Bug 11054: Specify utf8 encoding when creating a child record

When the only non-ascii characters in a new child record
are latin + diacritic in the range covered by Latin-1 they
are not being added to a new child record correctly encoded.
Explicitly setting the record's encodiing ensures that
they are interpreted correctly.

see "Unicode Bug" in perldoc perlunicode for background on
why these characters are special

Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com>
Works well. No koha-qa errors.

Same test as before
1) with a record with diacritics in author/title,
2) press new child record, on new record 773adt
with mangled accents
3) Apply patch
4) Repeat 2, now good results

Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de>
Passes all tests and QA script.
Tested with a record containing umlauts in the title,
$t was empty before and umlauts in $a mangled.
After applying the patch fields are correctly filled and
diacritics display correctly.
Comment 11 Galen Charlton 2013-10-31 23:23:47 UTC
Pushed to master.  Thanks, Colin!
Comment 12 Tomás Cohen Arazi (tcohen) 2013-11-20 16:44:51 UTC
This patch has been pushed to 3.12.x, will be in 3.12.7.

Thanks Colin!