Bug 6895

Summary: Diacritics in Pootle/po files are broken in source text
Product: Koha Reporter: Katrin Fischer <katrin.fischer>
Component: I18N/L10NAssignee: Chris Cormack <chris>
Status: CLOSED FIXED QA Contact: Bugs List <koha-bugs>
Severity: blocker    
Priority: P1 - high CC: chris, f.demians, koha.sekjal, paul.poulain
Version: 3.6   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 6752    
Bug Blocks:    
Attachments: pootle screenshot
Bug 6895 : First attempt at fixing the diacritics bug
signed off patch
Bug 6895 : First attempt at fixing the diacritics bug
Proposed patch
Bug 6895 Diacritics in Pootle/po files are broken in source text

Description Katrin Fischer 2011-09-21 10:07:02 UTC
Created attachment 5491 [details]
pootle screenshot

In Pootle diacritics in terms that are to be translated are broken. (See screenshot for example)
Comment 1 Chris Cormack 2011-10-05 02:10:45 UTC
Definitely a bug not a new feature, looking into this now, with the hope of getting it fixed before string freeze
Comment 2 Chris Cormack 2011-10-05 03:18:17 UTC
I have found the problem

binmode( OUTPUT, ":utf8" );

In xgettext.pl is causing the problems, without it though we get this

Wide character in print at ./xgettext.pl line 230.
Wide character in print at ./xgettext.pl line 230.
/tmp/chKkLCrt0N:53997:14: invalid multibyte sequence
/tmp/chKkLCrt0N:77820:9: invalid multibyte sequence
msgmerge: found 2 fatal errors

Will look into fixing the wide characters now
Comment 3 Chris Cormack 2011-10-05 03:41:21 UTC Comment hidden (obsolete)
Comment 4 Katrin Fischer 2011-10-05 06:33:41 UTC
I am testing this and a first test with 3.4.5 looks very promising.
I have updated the German po files using 'perl translate update de-DE'. Not a lot of strings need retranslating and all the 'candidates' like names with diacritics from the about page seem to show up in my list.
Comment 5 Katrin Fischer 2011-10-06 06:55:23 UTC Comment hidden (obsolete)
Comment 6 Paul Poulain 2011-10-12 15:38:26 UTC Comment hidden (obsolete)
Comment 7 Paul Poulain 2011-10-12 15:46:37 UTC
QA comment
  * it used to work in 3.2-, so it must have been caused by the switch to T::T. Investigating xgettext.pl history, it seems it has been added by a patch from chris, in april.
  * I don't know what kind of counter-patch should be sent, the patch switches back to the previous behaviour that was OK
  * diacritics in english version are almost only in about.pl, in the contributor list. It means they are not translated, but names does not need to be translated, so the problem is not a major one (that's why noone spotted it before)
  * why is there
-<p class="submit"><input id="submit" type="submit" value="Login &raquo;" tabindex="4" /></p>
+<p class="submit"><input id="submit" type="submit" value="Login" tabindex="4" /></p>
and
-<p class="submit"><input id="submit" type="submit" value="Login &raquo;" tabindex="4" /></p>
+<p class="submit"><input id="submit" type="submit" value="Login" tabindex="4" /></p>
in the patch ? If i'm not mistaking it's not related to this problem. (the macles.tt has probably be used for testing purposes.

Chris, please resubmit a patch removing those lines, i'll mark passed QA, as it fixes the problem.
Comment 8 Chris Cormack 2011-10-12 19:57:03 UTC
No those characters break the translation, along with the fixed diacritics, changing the to actual utf8 characters makes the string translatable.
Comment 9 Paul Poulain 2011-10-13 06:57:44 UTC
OK, passed QA then
Comment 10 Paul Poulain 2011-11-04 07:09:53 UTC
Patch pushed, please test (my 1st push!!!)

(cced the translation manager)
Comment 11 Katrin Fischer 2012-02-24 11:51:13 UTC
It seems this bug has reoccured in 3.6.4. The diacritics are broken again in the source files, requiring to be retranslated.
Comment 12 Frédéric Demians 2012-02-24 12:09:01 UTC Comment hidden (obsolete)
Comment 13 Chris Cormack 2012-02-24 19:59:06 UTC
(In reply to comment #12)
> Created attachment 7851 [details] [review]
> Proposed patch
> 
> Apply on HEAD/3.6.x
> Very needed for next 3.6.x (3.6.5)

I wonder how it crept back in? 
Ah ha!

Commit 
263dded8

For bug 6752

Yep this patch undoes the accidental regression.
Comment 14 Chris Cormack 2012-02-24 20:00:04 UTC
Created attachment 7865 [details] [review]
Bug 6895 Diacritics in Pootle/po files are broken in source text

To test:

  git checkout 3.6.x
  cd misc/translator
  ./tmpl_process3.pl update -i ../../koha-tmpl/opac-tmpl/prog/en/ \
    -s ./po/fr-FR-i-opac-t-prog-v-3006000.po -r

  po/fr-FR-i-opac-t-prog-v-3006000.po contains broken diacritics for NORMARC
  strings for example

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 15 Ian Walls 2012-02-24 20:19:46 UTC
Simple patch to fix one-line regression caused by blanket changes in bug 6752.  As noted on that bug, there could be such regressions, but we'd have time to catch them.  So here's catching.

Marking as Passed QA.