Bug 5883 - UTF-8 characters in items (952) get encoded wrong when the record is modified.
Summary: UTF-8 characters in items (952) get encoded wrong when the record is modified.
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: 3.4
Hardware: All All
: PATCH-Sent (DO NOT USE) normal (vote)
Assignee: Frédérick Capovilla
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2011-03-16 14:19 UTC by Frédérick Capovilla
Modified: 2013-12-05 20:04 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Prevent re-encoding the items if they are already in UTF-8 in ModBiblio. (1.39 KB, patch)
2011-03-21 18:45 UTC, Frédérick Capovilla
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Frédérick Capovilla 2011-03-16 14:19:04 UTC
If a record already contains contains items with UTF-8 characters (accented characters for example), those characters become garbled when the record is modified.

Example :
"méthode" in field 952$3 becomes "méthode" after the record is saved.
Comment 1 Frédérick Capovilla 2011-03-21 17:11:31 UTC
Additionnal informations :
The problem seems to happen when the character at offset 17 of the leader is a space. Koha puts a '7' there by default on new records.

Example leader where the problem applies : 26489nam a2203769 a 4500
Comment 2 Frédérick Capovilla 2011-03-21 18:13:34 UTC
(In reply to comment #1)
> Additionnal informations :
> The problem seems to happen when the character at offset 17 of the leader is a
> space. Koha puts a '7' there by default on new records.
> 
> Example leader where the problem applies : 26489nam a2203769 a 4500

After doing more tests, it seems the leader isn't the cause of the problem.

I found that using accented characters in the record when modifying it can cause the encoding problem on the items.

Still investigating...
Comment 3 Frédérick Capovilla 2011-03-21 18:45:29 UTC
Created attachment 3384 [details] [review]
Prevent re-encoding the items if they are already in UTF-8 in ModBiblio.

I think I have found the source of the problem :

In ModBiblio, the items were re-encoded in UTF-8 to fix some possible encoding mixing problems, but there was no check to see if the items were already in utf-8 before doing the re-encoding. This might have caused some encoding mixing problems in the case where the record and the item each contained utf-8 characters.

Patch attached. Please test.
Comment 4 Julian Maurice 2011-03-23 14:19:38 UTC
Signed off.
Comment 5 Chris Cormack 2011-03-24 00:26:41 UTC
Pushed, please test
Comment 6 MJ Ray (software.coop) 2011-04-01 09:42:30 UTC
This bug is mentioned in:
Bug 5883 : Prevent re-encoding	the items if they are already in UTF-8 in ModBiblio. http://lists.koha-community.org/pipermail/koha-patches/2011-March/014204.html
Comment 7 Frédérick Capovilla 2011-06-07 18:55:47 UTC
I think I made a mistake on this patch :
Encode::encode DEACTIVATES the utf-8 flag on a string, so checking if the flag is deactivated before doing the Encode::encode doesn't make any sense. I guess the problem was solved on my Koha instances because the re-encoding is never done.

Reopening this bug so a good solution to this problem may be found.
Maybe remove the subfield re-encoding altogether?
Comment 8 Jared Camins-Esakov 2012-12-31 00:25:09 UTC
There have been no further reports of problems so I am marking this bug resolved.