Bug 14078

Summary: converting from ISO5426 is not complete
Product: Koha Reporter: Fridolin Somers <fridolin.somers>
Component: CatalogingAssignee: Fridolin Somers <fridolin.somers>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: f.demians, kyle, liz, m.de.rooy, mtompset, tomascohen, ztajoli
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 14078 - converting from ISO5426 is not complete
Bug 14078 - converting from ISO5426 is not complete - followup1
sample records
Bug 14078 - converting from ISO5426 is not complete
Bug 14078 - converting from ISO5426 is not complete
Bug 14078 - converting from ISO5426 is not complete followup1
Bug 14078 - converting from ISO5426 is not complete
Bug 14078 - converting from ISO5426 is not complete - followup1

Description Fridolin Somers 2015-04-29 10:12:28 UTC
Conversion of MARC from ISO5426 is defined in C4::Charset::char_decode5426().
Each character or combined characters conversion is defined in a map.

This conversion is missing some cases and some actual conversions are odd.
Comment 1 Fridolin Somers 2015-04-29 10:30:19 UTC
I've found only one full documentation about this conversion :
http://www.gymel.com/charsets/MAB2.html

Is it correct ?
Comment 2 Fridolin Somers 2015-04-29 10:32:38 UTC Comment hidden (obsolete)
Comment 3 Fridolin Somers 2015-04-29 10:36:34 UTC Comment hidden (obsolete)
Comment 4 Fridolin Somers 2015-04-29 10:37:35 UTC
Patches are actually drafts so I do not set to "need signoff".

Looking for feedback.
Comment 5 Zeno Tajoli 2015-04-29 11:56:24 UTC
Hi Fridolin,

I confirm you that http://www.gymel.com/charsets/MAB2.html is a good source for ISO 5426
Comment 6 Fridolin Somers 2015-05-22 15:55:39 UTC
(In reply to Zeno Tajoli from comment #5)
> Hi Fridolin,
> 
> I confirm you that http://www.gymel.com/charsets/MAB2.html is a good source
> for ISO 5426

Thanks a lot.

I set has need signoff
Comment 7 Mark Tompsett 2015-06-14 04:58:01 UTC
Do you have some test data?
Comment 8 Fridolin Somers 2015-06-23 16:04:34 UTC
(In reply to M. Tompsett from comment #7)
> Do you have some test data?

Here are some records.
Comment 9 Fridolin Somers 2015-06-23 16:05:00 UTC
Created attachment 40534 [details]
sample records
Comment 10 Frédéric Demians 2015-10-27 09:54:54 UTC Comment hidden (obsolete)
Comment 11 Frédéric Demians 2015-10-27 09:57:22 UTC Comment hidden (obsolete)
Comment 12 Frédéric Demians 2015-10-27 09:58:03 UTC Comment hidden (obsolete)
Comment 13 Fridolin Somers 2015-11-13 11:34:54 UTC
An example contained in sample, the 38th record :

Without patch :
Mexique, le nouveau ¸tigre aztèque¸ ? Shannon K. O'Neil (0032-9304)

With patch :
Mexique, le nouveau ″tigre aztèque″ ? Shannon K. O'Neil (0032-9304)
Comment 14 Jonathan Druart 2015-11-13 11:39:10 UTC
Created attachment 44834 [details] [review]
Bug 14078 - converting from ISO5426 is not complete

Conversion of MARC from ISO5426 is defined in C4::Charset::char_decode5426().
Each character or combined characters conversion is defined in a map.

This patch changes some odd actual conversions.

In char_decode5426(), only characters between 0xC0 and 0xDF will be used for combining with following charater :
  ($char >= 0xC0 && $char <= 0xDF)
So conversion like "$chars{0x81d1}=0x00b0" will never be used.
Rules for "h with breve below" use combining with 0xf9 but looks like the correct caracter is 0xd5.

See http://www.gymel.com/charsets/MAB2.html

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 15 Jonathan Druart 2015-11-13 11:39:15 UTC
Created attachment 44835 [details] [review]
Bug 14078 - converting from ISO5426 is not complete - followup1

Conversion of MARC from ISO5426 is defined in C4::Charset::char_decode5426().
Each character or combined characters conversion is defined in a map.

This patch adds missing conversions.

See http://www.gymel.com/charsets/MAB2.html

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 16 Tomás Cohen Arazi 2015-11-16 15:57:50 UTC
Patches pushed to master.

Thanks Frido!
Comment 17 Frédéric Demians 2015-11-29 21:00:34 UTC
This patch has been pushed to 3.20.x, will be in 3.20.6.
Comment 18 Liz Rea 2016-01-06 21:08:30 UTC
Pushed to 3.18.13, and released.