Description
Chris Cormack
2010-05-21 00:48:28 UTC
Label printing doesn't work with any Cirillic characters (we try print patron cards) (tested on Debian sid, Koha 3.2.1, Font::TTF and Text::PDF::TTFont0 was installed). Created attachment 3950 [details] [review] Partial fix to reduce invalid pdf generation due to wide character errors. I'm on 3.02.07.000 with Debian squeeze and I'm facing the same problem with German umlautes aswell as Turkish Characters in the records. PDF are invalid when created. *** Bug 3400 has been marked as a duplicate of this bug. *** Created attachment 4291 [details] [review] Bug 2246 - (Partial) Label printing doesn't work with Unicode characters This patch provides a very partial fix for this bug in that it reduces the number of pdf generation failures due to a "wide character" error. It does not ensure that all unicode characters will print correctly as this is dependent upon many other issues mentioned in this bug and various posts to the developer list. What this code does is test to see if the pdf stream is utf8 encoded and if it is, explicitly declares it to be so. Unicode chars will still print incorrectly, but the pdf will be created and should open properly in whatever pdf reader. You may test this by adding any character with a diacritical to the title of a bib and then attempting to generate a label pdf with the title of that bib. Before the patch is applied the resulting pdf should contain an error mentioning a wide character issue. After the patch is applied, the pdf should be valid. No documentation changes are necessary as a result of this patch. This patch should be backported to 3.2.x. Created attachment 4292 [details] [review] Bug 2246 - (Partial) Label printing doesn't work with Unicode characters This patch provides a very partial fix for this bug in that it reduces the number of pdf generation failures due to a "wide character" error. It does not ensure that all unicode characters will print correctly as this is dependent upon many other issues mentioned in this bug and various posts to the developer list. What this code does is test to see if the pdf stream is utf8 encoded and if it is, explicitly declares it to be so. Unicode chars will still print incorrectly, but the pdf will be created and should open properly in whatever pdf reader. You may test this by adding any character with a diacritical to the title of a bib and then attempting to generate a label pdf with the title of that bib. Before the patch is applied the resulting pdf should contain an error mentioning a wide character issue. After the patch is applied, the pdf should be valid. No documentation changes are necessary as a result of this patch. This patch should be backported to 3.2.x. Created attachment 4293 [details] [review] [SIGNED-OFF] Bug 2246 - (Partial) Label printing doesn't work with Unicode characters This patch provides a very partial fix for this bug in that it reduces the number of pdf generation failures due to a "wide character" error. It does not ensure that all unicode characters will print correctly as this is dependent upon many other issues mentioned in this bug and various posts to the developer list. What this code does is test to see if the pdf stream is utf8 encoded and if it is, explicitly declares it to be so. Unicode chars will still print incorrectly, but the pdf will be created and should open properly in whatever pdf reader. You may test this by adding any character with a diacritical to the title of a bib and then attempting to generate a label pdf with the title of that bib. Before the patch is applied the resulting pdf should contain an error mentioning a wide character issue. After the patch is applied, the pdf should be valid. No documentation changes are necessary as a result of this patch. This patch should be backported to 3.2.x. Printing German umlauts works without problem for me. I could reproduce the desribed behaviour with Cyrillic "Россия". Confirming that the signed-off patch does indeed allow the PDF to be opened, even if the Unicode characters are not properly displaying (as is the case for the Hindi word दिशा) or come with post-character cruft (as with author Brontë) Marking as Passed QA, as although this bug is far from resolved, this patch greatly improves the usability of the label creator for the occasions when only a few titles have non-ASCII characters. Pushed to master, please test *** Bug 6899 has been marked as a duplicate of this bug. *** Improvement With hopes of getting diacritics ( u dieresis and such) printing for my barcode labels, I dug into /home/koha344/koha/intranet/cgi-bin/labels/label-create-pdf.pl I found that if I set the utf8 flag off for the $line string, it would end up sending FC instead of C3BC in the PDF. So for the "standard PDF encodings that we currently allow (Helvitica, Courier, etc) the single byte maps to those common foreign characters just fine. It worked for me! Here is the "utf8::downgrade($line);" that I used from this section of code. I am running perl 5.12.1 (that I built), with the latest cpan modules pulled as of Sept 2011, and koha 3.4.4. (your wide character fix in PDF.pm helped). It seems to get through my latest version of PDF::Reuse, that I have installed. sub _print_text { my $label_text = shift; foreach my $text_line (@$label_text) { my $pdf_font = $pdf->Font($text_line->{'font'}); my $line = "BT /$pdf_font $text_line->{'font_size'} Tf $text_line->{'text_llx'} $text_line->{'text_lly'} Td ($text_line->{'line'}) Tj ET"; utf8::downgrade($line); # This forces the utf8 flag off so the single byte will pass thru for regular PDFEncoding etc. # (So now my basic diacritics in the standard character sets will not get double encoded. Yeah! # Note: This is not meant to deal with the case where we want to use unicode with TTF - truetype fonts. $pdf->Add($line); } } This bug is log overdue, and it might be possible for someone to test and patch it, since I might not be able to get it done before Oct 8th 2011 for the 3.6 release. Note: I didn't want to loose this info. I also tried this fix in my production 3.0.2 environment. I had to add the PDF.pm patch for wide chars, to get a PDF to be generated. Once I added the downgrade line in 3.0.2, my create-label-pdf.pl hung, with some recursion error and cpu at 100%. So I gave up on 3.0.2. The PDF:Reuse in that version might be the factor, or something else. I'm going to grab koha 3.4.5 and try the fix on that. We can't wait till next year for this, can we? I also coded it with a TTF truetype font, but was missing the description lines, so I didn't have time to figure out the drawing logic. We would need to add a way to select a font path, or upload one, and store it in the database. Also, this fix might be usable for other create-*-pdf.pl scripts (like patron cards). This is a partial fix if we need to support other languages. I tried making wajasu's edit to a system known to be having this problem--code updated as of last night at HEAD, and it did not resolve the problem for me--I got a "this PDF is broken and cannot be repaired" error. :( Created attachment 5713 [details] [review] Bug 2246 - (Partial) Map multibyte UTF8 to single byte for ISOLatin1 fonts (fixes diacritics <ASCII 256 decimal) Bug 2246 - (Partial) Map multibyte UTF8 to single byte for ISOLatin1 fonts (fixes diacritics <ASCII 256 decimal) This is a partial fix as well, which attempts to convert the internal representation of multibye UTF8 characters to their single byte in the native encoding (Latin-1) and allow them to pass through to the PDF stream. This ONLY fixes those ISOLatin1 diacritics and probably won't solve a full foreign language need. It probably solves the printing case for many historical records, but won't take care of the need for the interational community. I believe we need to use a full unicode embedded font in the final solution. Refer to utf8::downgrade($string,FAIL_OK); (see core perl /usr/share/perl5/core_perl/utf8.pm) Test: a) I selected a biblio that had a udiaresis (u with 2 dots above, i.e. Jurgen Habermas) and created a batch with the one record. I used the standard Helvitica font (no truetype) I exported it as a PDF and saw my label had "J (captial A tilde on top, 1/4) rgen" for the author. b) Applied patch. c) Exported the pdf again, and saw "J (u with two dots above) rgen" To see what changed in the PDF that was generated: a) Edit the label-create-pdf.pl and temporarily comment out the $pdf->Compress(1) line so that you can see the PDF test instructions when generating. (Export it.) b) Use a hexfile viewer (I use hexedit), and search for Habermas, and you will see the corresponding Jurgen with two bytes C3BC before the patch, and FC after the patch. (You can use od -x in unix to also view the PDF if you don't have hexedit). Some explanation: The utf8 flag is turned off, and the FC is passed thru. I tried Encode:decode routines, but I think they keep the perl internal utf8 flag on, and the bytes stream out as C3BC. I've read when strings are concatenated, the flag can switch on/off, so I hoped something in the PDF::Reuse module would not tern it back on (if thats what is helping). Observations: I tried this on my production 3.2 ish koha, and it did't work so this patch is dependent on other fixes such as http://bugs.koha-community.org/bugzilla3/attachment.cgi?id=4293 I hoped folks in older versions could make the change in production without an upgrade, but one could try as see since its a staff client tool. It worked for me 3.4.4 and 3.5.x koho git master as of October 5th 2011 Things that might be needed: Pertinent modules? MyTestEnv version HEAD Required PDF::API2 2.019 2 Yes PDF::API2::Page 2.019 2 Yes PDF::API2::Simple 1.1.4 1 Yes PDF::API2::Util 2.019 2 Yes PDF::Reuse 0.35 0.33 Yes PDF::Reuse::Barcode 0.05 0.05 Yes PDF::Table 0.9.3 0.9.3 Yes Unicode::Normalize 1.03 0.32 Yes perl -v This is perl 5, version 12, subversion 1 (v5.12.1) built for x86_64-linux If you test, be sure to test with a diacritic that has a corresponding ISOLatin1 mapping. I am a bit confused about your example - because I had tested German umlauts like 'Jürgen' with one of the last patches that got into master for this bug and it worked nicely. Could your problem here be related to unicode normalization? (In reply to comment #15) > I am a bit confused about your example - because I had tested German umlauts > like 'Jürgen' with one of the last patches that got into master for this bug > and it worked nicely. Could your problem here be related to unicode > normalization? It could be because of unicode normalization. I faintly recall making a decision to call or not call some unicode normalization routine when I wrote a program to convert to marcxml records from a legacy library syste 3 years ago. It might have been a windows cp1252 encoding originally, with characters mapping to utf8. I did this query against my database for diacritics: With mysql I run: select author,HEX(author),CHAR_LENGTH(author) from biblio where HEX(author) like '%C3BC%'; and get lots of records like: Moltmann, Jürgen. | 4D6F6C746D616E6E2C204AC3BC7267656E2E | 17 where the C3BC represents ü. The C3BC is what get written to the PDF file with out my patch. Maybe you can use a hex editor and see if your label export has C3BC. If so it might be the PDF reader that is able to map that. If yours is F9, then there must be a difference in the perl version/compile options or module code. Also, I saved a biblio as marcxml, and the xml contained JX;rgen in subfield code "a", so thats an XML entity. Saving marc.utf8 it has JuCC88rgen in the text areao of my hex editor. Another specific query I did for my test record. select author,HEX(author),CHAR_LENGTH(author) from biblio where author like '%Habermas%'; mysql version: 5.5.16 mysql> SHOW SESSION VARIABLES LIKE 'character_set%'; +--------------------------+----------------------------+ | Variable_name | Value | +--------------------------+----------------------------+ | character_set_client | utf8 | | character_set_connection | utf8 | | character_set_database | utf8 | | character_set_filesystem | binary | | character_set_results | utf8 | | character_set_server | utf8 | | character_set_system | utf8 | | character_sets_dir | /usr/share/mysql/charsets/ | +--------------------------+----------------------------+ Note: These are barcode labels I'm printing. I created a debian virtual machine dev environment as described by the community wiki, and see that perl 5.10.1 is what is used in debian squeeze. I applied my one liner patch (utf8::downgrade($line) in label-create-pdf.pl) and it had no effect. So I suspect some change in perl, or even a side effect of how certain modules behave is the reason. I'm noting this so a future upgrade might fix the issue for someone if its needed. I've been running perl 5.12.1, and now 5.14.2 is out. - Oct 24 2011 I'm marking these label/diacritical related bugs as WONTFIX due to the well rehearsed issues of the pdf standard and diacriticals. If someone wants to put time into this, feel free to reopen and take ownership of these bugs. *** Bug 8563 has been marked as a duplicate of this bug. *** *** Bug 13627 has been marked as a duplicate of this bug. *** |