When you select New child record from the catalogue detail page. If the data that will be used to create the host field contains latin characters with diacritics from the range covered by latin-1 they are incorrectly encoded in the resulting child record. This is not the case with non-latin characters (Cyrillic, Arabic and Chinese tested) and I suspect with latin diacritic combinations that are not in the latin-1 range
Created attachment 22005 [details] [review] Proposed Patch To test. Find a record with an accented character in the title e.g. a vowel with an acute or grave or an n with a tilde in the detail page select new/new child record the title will be copied into 773 $t (marc21) without the patch garbage replaces the accented characters with the patch they appear correctly. I've confirmed that the change does not have any detrimental effects when copying non-latin characters e.g. Chinese, Cyrillic, Arabic which were already being correctly copied
Created attachment 22006 [details] [review] [SIGNED-OFF] Bug 11054: Specify utf8 encoding when creating a child record When the only non-ascii characters in a new child record are latin + diacritic in the range covered by Latin-1 they are not being added to a new child record correctly encoded. Explicitly setting the record's encodiing ensures that they are interpreted correctly. see "Unicode Bug" in perldoc perlunicode for background on why these characters are special Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Work as described. No koha-qa errors Test search for a record with diacritics on author or title press New Child Record without patch, 773's with mangled chars with patch, all fine This has nothing to do with major problems of encoding, here we create a new record and as CPAN page states, default encoding is MARC-8, so it's the right solution.
QA comment * one-liner * I'm wondering if it would not be better to use C4/Charset.pm sub SetMarcUnicodeFlag($marc_record, $marc_flavour); for this : it raises the utf8 flag, but ALSO set UNIMARC field 100 if applicable or position 9 of the leader for MARC21/NORMARC Setting to "in discussion", colin, argue and/or reset status to signed-off
Paul SetMarcUnicodeFlag sets the MARC flag in the leader the bug is not related to this the patch is setting the utf-8 flag in the perl var containing the string. A different thing Setting back to signed off
(In reply to Colin Campbell from comment #4) > Paul SetMarcUnicodeFlag sets the MARC flag in the leader the bug is not > related to this the patch is setting the utf-8 flag in the perl var > containing the string. A different thing > Setting back to signed off Right, but it ALSO set the flag: $marc_record->encoding('UTF-8'); (3 lines after the sub prototype) I still think we should use this sub instead of a direct ->encoding
For what is worth, with patch applied, I replaced line 824 in addbiblio.pl $record->encoding( 'UTF-8' ); with SetMarcUnicodeFlag($record, $marcflavour); Same (good) result, and perhaps is more general.
Created attachment 22317 [details] [review] Amended patch Make the change as suggested in the comments
Switched back to needs sign off. Bernardo it should pass your test
Created attachment 22318 [details] [review] [SIGNED-OFF] Bug 11054: Specify utf8 encoding when creating a child record When the only non-ascii characters in a new child record are latin + diacritic in the range covered by Latin-1 they are not being added to a new child record correctly encoded. Explicitly setting the record's encodiing ensures that they are interpreted correctly. see "Unicode Bug" in perldoc perlunicode for background on why these characters are special Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works well. No koha-qa errors. Same test as before 1) with a record with diacritics in author/title, 2) press new child record, on new record 773adt with mangled accents 3) Apply patch 4) Repeat 2, now good results
Created attachment 22449 [details] [review] [PASSED QA] Bug 11054: Specify utf8 encoding when creating a child record When the only non-ascii characters in a new child record are latin + diacritic in the range covered by Latin-1 they are not being added to a new child record correctly encoded. Explicitly setting the record's encodiing ensures that they are interpreted correctly. see "Unicode Bug" in perldoc perlunicode for background on why these characters are special Signed-off-by: Bernardo Gonzalez Kriegel <bgkriegel@gmail.com> Works well. No koha-qa errors. Same test as before 1) with a record with diacritics in author/title, 2) press new child record, on new record 773adt with mangled accents 3) Apply patch 4) Repeat 2, now good results Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes all tests and QA script. Tested with a record containing umlauts in the title, $t was empty before and umlauts in $a mangled. After applying the patch fields are correctly filled and diacritics display correctly.
Pushed to master. Thanks, Colin!
This patch has been pushed to 3.12.x, will be in 3.12.7. Thanks Colin!