Bug 6895 - Diacritics in Pootle/po files are broken in source text
Summary: Diacritics in Pootle/po files are broken in source text
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: 3.6
Hardware: All All
: P1 - high blocker (vote)
Assignee: Chris Cormack
QA Contact: Bugs List
URL:
Keywords:
Depends on: 6752
Blocks:
  Show dependency treegraph
 
Reported: 2011-09-21 10:07 UTC by Katrin Fischer
Modified: 2019-06-27 09:24 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
pootle screenshot (18.88 KB, image/png)
2011-09-21 10:07 UTC, Katrin Fischer
Details
Bug 6895 : First attempt at fixing the diacritics bug (3.40 KB, patch)
2011-10-05 03:41 UTC, Chris Cormack
Details | Diff | Splinter Review
signed off patch (3.56 KB, patch)
2011-10-06 06:55 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 6895 : First attempt at fixing the diacritics bug (3.61 KB, patch)
2011-10-12 15:38 UTC, Paul Poulain
Details | Diff | Splinter Review
Proposed patch (1.12 KB, patch)
2012-02-24 12:09 UTC, Frédéric Demians
Details | Diff | Splinter Review
Bug 6895 Diacritics in Pootle/po files are broken in source text (1.18 KB, patch)
2012-02-24 20:00 UTC, Chris Cormack
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Katrin Fischer 2011-09-21 10:07:02 UTC
Created attachment 5491 [details]
pootle screenshot

In Pootle diacritics in terms that are to be translated are broken. (See screenshot for example)
Comment 1 Chris Cormack 2011-10-05 02:10:45 UTC
Definitely a bug not a new feature, looking into this now, with the hope of getting it fixed before string freeze
Comment 2 Chris Cormack 2011-10-05 03:18:17 UTC
I have found the problem

binmode( OUTPUT, ":utf8" );

In xgettext.pl is causing the problems, without it though we get this

Wide character in print at ./xgettext.pl line 230.
Wide character in print at ./xgettext.pl line 230.
/tmp/chKkLCrt0N:53997:14: invalid multibyte sequence
/tmp/chKkLCrt0N:77820:9: invalid multibyte sequence
msgmerge: found 2 fatal errors

Will look into fixing the wide characters now
Comment 3 Chris Cormack 2011-10-05 03:41:21 UTC Comment hidden (obsolete)
Comment 4 Katrin Fischer 2011-10-05 06:33:41 UTC
I am testing this and a first test with 3.4.5 looks very promising.
I have updated the German po files using 'perl translate update de-DE'. Not a lot of strings need retranslating and all the 'candidates' like names with diacritics from the about page seem to show up in my list.
Comment 5 Katrin Fischer 2011-10-06 06:55:23 UTC Comment hidden (obsolete)
Comment 6 Paul Poulain 2011-10-12 15:38:26 UTC Comment hidden (obsolete)
Comment 7 Paul Poulain 2011-10-12 15:46:37 UTC
QA comment
  * it used to work in 3.2-, so it must have been caused by the switch to T::T. Investigating xgettext.pl history, it seems it has been added by a patch from chris, in april.
  * I don't know what kind of counter-patch should be sent, the patch switches back to the previous behaviour that was OK
  * diacritics in english version are almost only in about.pl, in the contributor list. It means they are not translated, but names does not need to be translated, so the problem is not a major one (that's why noone spotted it before)
  * why is there
-<p class="submit"><input id="submit" type="submit" value="Login &raquo;" tabindex="4" /></p>
+<p class="submit"><input id="submit" type="submit" value="Login" tabindex="4" /></p>
and
-<p class="submit"><input id="submit" type="submit" value="Login &raquo;" tabindex="4" /></p>
+<p class="submit"><input id="submit" type="submit" value="Login" tabindex="4" /></p>
in the patch ? If i'm not mistaking it's not related to this problem. (the macles.tt has probably be used for testing purposes.

Chris, please resubmit a patch removing those lines, i'll mark passed QA, as it fixes the problem.
Comment 8 Chris Cormack 2011-10-12 19:57:03 UTC
No those characters break the translation, along with the fixed diacritics, changing the to actual utf8 characters makes the string translatable.
Comment 9 Paul Poulain 2011-10-13 06:57:44 UTC
OK, passed QA then
Comment 10 Paul Poulain 2011-11-04 07:09:53 UTC
Patch pushed, please test (my 1st push!!!)

(cced the translation manager)
Comment 11 Katrin Fischer 2012-02-24 11:51:13 UTC
It seems this bug has reoccured in 3.6.4. The diacritics are broken again in the source files, requiring to be retranslated.
Comment 12 Frédéric Demians 2012-02-24 12:09:01 UTC Comment hidden (obsolete)
Comment 13 Chris Cormack 2012-02-24 19:59:06 UTC
(In reply to comment #12)
> Created attachment 7851 [details] [review]
> Proposed patch
> 
> Apply on HEAD/3.6.x
> Very needed for next 3.6.x (3.6.5)

I wonder how it crept back in? 
Ah ha!

Commit 
263dded8

For bug 6752

Yep this patch undoes the accidental regression.
Comment 14 Chris Cormack 2012-02-24 20:00:04 UTC
Created attachment 7865 [details] [review]
Bug 6895 Diacritics in Pootle/po files are broken in source text

To test:

  git checkout 3.6.x
  cd misc/translator
  ./tmpl_process3.pl update -i ../../koha-tmpl/opac-tmpl/prog/en/ \
    -s ./po/fr-FR-i-opac-t-prog-v-3006000.po -r

  po/fr-FR-i-opac-t-prog-v-3006000.po contains broken diacritics for NORMARC
  strings for example

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 15 Ian Walls 2012-02-24 20:19:46 UTC
Simple patch to fix one-line regression caused by blanket changes in bug 6752.  As noted on that bug, there could be such regressions, but we'd have time to catch them.  So here's catching.

Marking as Passed QA.