Bug 19084 - Spine label with BN_IN UTF8 data rendered incorrectly
Summary: Spine label with BN_IN UTF8 data rendered incorrectly
Status: BLOCKED
Alias: None
Product: Koha
Classification: Unclassified
Component: Label/patron card printing (show other bugs)
Version: Main
Hardware: All All
: P5 - low major (vote)
Assignee: Chris Nighswonger
QA Contact: Testopia
URL: https://rt.cpan.org/Ticket/Display.ht...
Keywords:
Depends on:
Blocks:
 
Reported: 2017-08-11 07:16 UTC by Indranil Das Gupta
Modified: 2020-01-11 21:10 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
View of expected output vs output rendered (1.96 KB, image/png)
2017-08-11 07:16 UTC, Indranil Das Gupta
Details

Note You need to log in before you can comment on or make changes to this bug.
Description Indranil Das Gupta 2017-08-11 07:16:39 UTC
Created attachment 65878 [details]
View of expected output vs output rendered

I was trying to generate spine labels with call numbers in my mother tongue Bengali, which is spoken by 250 million people making it the 7th largest spoken language in the world.

In the attachment, the text bordered with green outline is how it is expected to appear, however what appears in the PDF is the text within the red border on left. FWIW, the koha-conf.xml ttf settings for Bengali font are in place, checked with different fonts, including Lohit Bengali which is the default BN_IN font on RHEL, Fedora and Ubuntu etc. as well as with NOTO Sans Bengali from Google.

None of the conjunct clusters as well as  matraas that go before a letter are rendered in the correct order. Unless this is fixed Koha can not be used to correctly generate spine labels for Bengali books 

For reference, here is discussion about a similar problem with iText - http://palashray.com/making-itext-work-with-indic-scripts/
Comment 1 Chris Nighswonger 2017-08-14 14:24:21 UTC
Quoting from the mailing list:

This problem seems to be present for most Indian languages whenever
they have conjunct clusters in their call numbers (depicted as
grapheme clusters in an unicoded string).

To describe the problem simply - the order of chars rendered is
incorrect in the output. For example the string - "শেখর" is
represented by the following code points -
\x{09B6}\x{09C7}\x{0996}\x{09B0}.

Now here is the catch: \x{09B6} represents the bengali letter SHA,
whereas \x{09C7} represents the bengali vowel sign E; however in the
correct linguistic visual presentation, the vowel sign E sits before
the SHA, which is not how the codepoints are arranged in the unicode
string.

I looked around PDF::Reuse, Text::PDF::TTFont etc modules, what seems
to me to be the root of this problem is the unpacku() method which is
pushing the unicode characters into an array in order to introduce
them into the PDF content stream with the correct font information.
However, being pushed in in that order, I think may be the cause of
this problem, which would make this an upstream issue rather than a
Koha bug.

cheers
indranil
Comment 2 Chris Nighswonger 2017-08-14 14:36:55 UTC
I wonder if this problem also occurs in other abugida writing systems?
Comment 3 Indranil Das Gupta 2017-08-14 20:51:24 UTC
Yes! In fact, I was perhaps hasty in trashing unpacku() method. The root of the trouble is the out_text() method where the the actual glyphs are parsed into the PDF content stream. What is happening here is that the individual codepoints pushed into @clist by unpacku() are being listed out one at a time into the PDF content stream as glyphs, *without* the necessary glyph reordering taking place.

So I would expect every single abiguda writing system to be be impacted.
Comment 4 Chris Nighswonger 2017-08-15 19:37:06 UTC
(In reply to Indranil Das Gupta from comment #3)
> What is happening here is that the individual
> codepoints pushed into @clist by unpacku() are being listed out one at a
> time into the PDF content stream as glyphs, *without* the necessary glyph
> reordering taking place.

It would seem that the glyph order should never be "changed" in the first place. ie. the order they are supplied should be preserved throughout the entire process of PDF generation.
Comment 5 Indranil Das Gupta 2017-08-15 19:44:30 UTC
Hi Chris,

The example I referred to on the m/l has the following codepoint order \x{09B6}\x{09C7}\x{0996}\x{09B0} and that's exactly how PDF::Reuse and PDI::API2 is pushing it out.

However as per rules of glyph reordered necessity by Bengali, the actual ordering of glyphs (as opposed to the codepoint order) should be \x{09C7}\x{09B6}\x{0996}\x{09B0}.

LibreOffice which uses the ICU rules, handles this perfectly within ODF as well as during the PDF export, as does any software that uses Pango as the rendering backend.

basically calls need to be made to pick up the correct information from the GSUB and GPOS tables of the font being embedded, which this two perl libs apparently (from my limited reading so far) do not do.
Comment 6 Chris Nighswonger 2017-08-15 19:46:58 UTC
This Koha bug depends on this CPAN bug:

https://rt.cpan.org/Ticket/Display.html?id=122778
Comment 7 Josef Moravec 2017-08-25 09:35:48 UTC
Marking blocked because of comment 6