Bug 1634

Summary: Incorrect encoding convertion in MARC record form
Product: Koha Reporter: Chris Cormack <chris>
Component: CatalogingAssignee: Galen Charlton <gmcharlt>
Status: CLOSED REMIND QA Contact:
Severity: normal    
Priority: P3    
Version: 2.2.9   
Hardware: PC   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: This is screenshot of the bug
That's how should be
patch
Framework structure with cyrillic symbols
MARC record
The title of the book in UTF-8

Description Chris Cormack 2010-05-21 00:32:51 UTC


---- Reported by sanyok_ko@mail.ru 2007-11-30 02:47:21 ----

I don't know english, so see screenshots and patch :)


Koha version:  	2.2.9
OS version: 	2.6.11-1.1369_FC4smp on i386 GNU/Linux
Perl version: 	5.008006
MySQL version: 	mysql Ver 14.7 Distrib 4.1.20, for redhat-linux-gnu (i386) using readline 4.3
Apache version: Apache/2.0.54 Server built: May 23 2005 08:12:24



---- Additional Comments From sanyok_ko@mail.ru 2007-11-30 02:48:16 ----

Created an attachment
This screenstot of bug





---- Additional Comments From sanyok_ko@mail.ru 2007-11-30 02:49:58 ----

Created an attachment
That's how should be





---- Additional Comments From sanyok_ko@mail.ru 2007-11-30 02:50:34 ----

Created an attachment
patch





---- Additional Comments From gmcharlt@gmail.com 2007-11-30 07:49:34 ----

Thanks -- I'm reviewing the screenshots and patch now.

I also want to check to see if this is a problem with Koha 3 -- to that end, could you attach the example MARC record to the bug.



---- Additional Comments From paul.poulain@biblibre.com 2007-11-30 08:19:24 ----

galen, just to let you know : one of our customers, that uses koha 3 has some problems that seems like this one.
But I never could reproduce it myself.

Then I realised the library has mySQL 4.1 and I have mySQL 5.0 Not sure it's the source of the problem, but maybe.



---- Additional Comments From sanyok_ko@mail.ru 2007-12-03 00:47:31 ----

Created an attachment
Framework structure with cyrillic symbols





---- Additional Comments From sanyok_ko@mail.ru 2007-12-03 00:48:09 ----

Created an attachment
MARC record





---- Additional Comments From jmf@liblime.com 2007-12-04 09:00:24 ----

(In reply to comment #7)
> Created an attachment [edit]
> MARC record
> 

Thanks for this record, however, please note that it is NOT in UNICODE/UTF-8, or at least it's not indicated in the record. If it were, the LEADER/09 would be set to 'a'. Please let us know the intended encoded of this record and if possible, resubmit as UNICODE/UTF-8 with LEADER/09 set to 'a'.



---- Additional Comments From sanyok_ko@mail.ru 2007-12-04 14:13:46 ----

This MARC record was produced by Koha export utility. Sorry, but I don't know MARC record format, and can't set the LEADER/09 to 'a'.

Using this steps, you can check, that this MARC record is in UNICODE/UTF-8:
1. Rename Koha.mrc into Koha.txt
2. Open Koha.txt with your web browser (I used Mozilla/Firefox 2.0.0.11 on Windows)
3. Set browser display encoding to UNICODE/UTF8
4. Compare text in the browser with the screenshot 'That's how should be'
5. You can find the title of the book by string 'aПриемы' with find in page dialog and copy/paste technics.

You can check the encoding of this record using Cyrillic UTF8 encoding table (http://www.utf8-chartable.de/unicode-utf8-table.pl?start=1024)
Just compare some first letters of the title.

This is title of the book in UTF-8 encoding
[root@server1 ~]# echo "Приемы объектно-ориентированного проектирования." | hexdump
0000000 9fd0 80d1 b8d0 b5d0 bcd0 8bd1 d020 d0be
0000010 d1b1 d08a d0b5 d1ba d082 d0bd 2dbe bed0
0000020 80d1 b8d0 b5d0 bdd0 82d1 b8d0 80d1 bed0
0000030 b2d0 b0d0 bdd0 bdd0 bed0 b3d0 bed0 d020
0000040 d1bf d080 d0be d0b5 d1ba d082 d1b8 d080
0000050 d0be d0b2 d0b0 d0bd d1b8 2e8f 000a
000005d

Additionally, I have found, that this linux commands will produce the same malformed title, as you can see at screenshot:
echo 'title of the book in UTF8' | iconv -f latin-1 -t utf-8

[root@server1 ~]# echo "Приемы объектно-ориентированного проектирования." | iconv -f latin1 -t utf-8 | hexdump
0000000 90c3 9fc2 91c3 80c2 90c3 b8c2 90c3 b5c2
0000010 90c3 bcc2 91c3 8bc2 c320 c290 c3be c290
0000020 c3b1 c291 c38a c290 c3b5 c290 c3ba c291
0000030 c382 c290 c3bd c290 2dbe 90c3 bec2 91c3
0000040 80c2 90c3 b8c2 90c3 b5c2 90c3 bdc2 91c3
0000050 82c2 90c3 b8c2 91c3 80c2 90c3 bec2 90c3
0000060 b2c2 90c3 b0c2 90c3 bdc2 90c3 bdc2 90c3
0000070 bec2 90c3 b3c2 90c3 bec2 c320 c290 c3bf
0000080 c291 c380 c290 c3be c290 c3b5 c290 c3ba
0000090 c291 c382 c290 c3b8 c291 c380 c290 c3be
00000a0 c290 c3b2 c290 c3b0 c290 c3bd c290 c3b8
00000b0 c291 2e8f 000a
00000b5



---- Additional Comments From sanyok_ko@mail.ru 2007-12-04 14:42:03 ----

Created an attachment
The title of the book in UTF-8





---- Additional Comments From nengard@gmail.com 2009-12-23 17:58:40 ----

This bug hasn't been touched in nearly a year, please revisit and test and close if appropriate.



--- Bug imported by chris@bigballofwax.co.nz 2010-05-21 00:32 UTC  ---

This bug was previously known as _bug_ 1634 at http://bugs.koha.org/cgi-bin/bugzilla3/show_bug.cgi?id=1634
Imported an attachment (id=356)
Imported an attachment (id=357)
Imported an attachment (id=358)
Imported an attachment (id=359)
Imported an attachment (id=360)
Imported an attachment (id=361)

Actual time not defined. Setting to 0.0
The original reporter of this bug does not have
   an account here. Reassigning to the person who moved
   it here: chris@bigballofwax.co.nz.
   Previous reporter was sanyok_ko@mail.ru.
The original submitter of attachment 356 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 357 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 358 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 359 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 360 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.
The original submitter of attachment 361 [details] is unknown.
   Reassigning to the person who moved it here: chris@bigballofwax.co.nz.