Bug 14078 - converting from ISO5426 is not complete
Summary: converting from ISO5426 is not complete
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Fridolin SOMERS
QA Contact: Testopia
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2015-04-29 10:12 UTC by Fridolin SOMERS
Modified: 2017-06-14 22:10 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 14078 - converting from ISO5426 is not complete (1.76 KB, patch)
2015-04-29 10:32 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
Bug 14078 - converting from ISO5426 is not complete - followup1 (2.74 KB, patch)
2015-04-29 10:36 UTC, Fridolin SOMERS
Details | Diff | Splinter Review
sample records (155.65 KB, application/download)
2015-06-23 16:05 UTC, Fridolin SOMERS
Details
Bug 14078 - converting from ISO5426 is not complete (1.82 KB, patch)
2015-10-27 09:54 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 14078 - converting from ISO5426 is not complete (1.82 KB, patch)
2015-10-27 09:57 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 14078 - converting from ISO5426 is not complete followup1 (2.79 KB, patch)
2015-10-27 09:58 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 14078 - converting from ISO5426 is not complete (1.89 KB, patch)
2015-11-13 11:39 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 14078 - converting from ISO5426 is not complete - followup1 (2.86 KB, patch)
2015-11-13 11:39 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin SOMERS 2015-04-29 10:12:28 UTC
Conversion of MARC from ISO5426 is defined in C4::Charset::char_decode5426().
Each character or combined characters conversion is defined in a map.

This conversion is missing some cases and some actual conversions are odd.
Comment 1 Fridolin SOMERS 2015-04-29 10:30:19 UTC
I've found only one full documentation about this conversion :
http://www.gymel.com/charsets/MAB2.html

Is it correct ?
Comment 2 Fridolin SOMERS 2015-04-29 10:32:38 UTC Comment hidden (obsolete)
Comment 3 Fridolin SOMERS 2015-04-29 10:36:34 UTC Comment hidden (obsolete)
Comment 4 Fridolin SOMERS 2015-04-29 10:37:35 UTC
Patches are actually drafts so I do not set to "need signoff".

Looking for feedback.
Comment 5 Zeno Tajoli 2015-04-29 11:56:24 UTC
Hi Fridolin,

I confirm you that http://www.gymel.com/charsets/MAB2.html is a good source for ISO 5426
Comment 6 Fridolin SOMERS 2015-05-22 15:55:39 UTC
(In reply to Zeno Tajoli from comment #5)
> Hi Fridolin,
> 
> I confirm you that http://www.gymel.com/charsets/MAB2.html is a good source
> for ISO 5426

Thanks a lot.

I set has need signoff
Comment 7 M. Tompsett 2015-06-14 04:58:01 UTC
Do you have some test data?
Comment 8 Fridolin SOMERS 2015-06-23 16:04:34 UTC
(In reply to M. Tompsett from comment #7)
> Do you have some test data?

Here are some records.
Comment 9 Fridolin SOMERS 2015-06-23 16:05:00 UTC
Created attachment 40534 [details]
sample records
Comment 10 Frédéric Demians 2015-10-27 09:54:54 UTC Comment hidden (obsolete)
Comment 11 Frédéric Demians 2015-10-27 09:57:22 UTC Comment hidden (obsolete)
Comment 12 Frédéric Demians 2015-10-27 09:58:03 UTC Comment hidden (obsolete)
Comment 13 Fridolin SOMERS 2015-11-13 11:34:54 UTC
An example contained in sample, the 38th record :

Without patch :
Mexique, le nouveau ¸tigre aztèque¸ ? Shannon K. O'Neil (0032-9304)

With patch :
Mexique, le nouveau ″tigre aztèque″ ? Shannon K. O'Neil (0032-9304)
Comment 14 Jonathan Druart 2015-11-13 11:39:10 UTC
Created attachment 44834 [details] [review]
Bug 14078 - converting from ISO5426 is not complete

Conversion of MARC from ISO5426 is defined in C4::Charset::char_decode5426().
Each character or combined characters conversion is defined in a map.

This patch changes some odd actual conversions.

In char_decode5426(), only characters between 0xC0 and 0xDF will be used for combining with following charater :
  ($char >= 0xC0 && $char <= 0xDF)
So conversion like "$chars{0x81d1}=0x00b0" will never be used.
Rules for "h with breve below" use combining with 0xf9 but looks like the correct caracter is 0xd5.

See http://www.gymel.com/charsets/MAB2.html

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 15 Jonathan Druart 2015-11-13 11:39:15 UTC
Created attachment 44835 [details] [review]
Bug 14078 - converting from ISO5426 is not complete - followup1

Conversion of MARC from ISO5426 is defined in C4::Charset::char_decode5426().
Each character or combined characters conversion is defined in a map.

This patch adds missing conversions.

See http://www.gymel.com/charsets/MAB2.html

Signed-off-by: Frederic Demians <f.demians@tamil.fr>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 16 Tomás Cohen Arazi 2015-11-16 15:57:50 UTC
Patches pushed to master.

Thanks Frido!
Comment 17 Frédéric Demians 2015-11-29 21:00:34 UTC
This patch has been pushed to 3.20.x, will be in 3.20.6.
Comment 18 Liz Rea 2016-01-06 21:08:30 UTC
Pushed to 3.18.13, and released.