Bug 8044

Summary: Localization for Perl scripts and modules
Product: Koha Reporter: Julian Maurice <julian.maurice>
Component: I18N/L10NAssignee: Julian Maurice <julian.maurice>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: enhancement    
Priority: P5 - low CC: adrien.saurat, chris, f.demians, fridolin.somers, gmcharlt, jonathan.druart, julian.maurice, katrin.fischer, m.de.rooy, magnus, ruth
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on:    
Bug Blocks: 11904, 11848, 22050    
Attachments: First draft for Koha localization using Locale::Maketext
Translate basket CSV column names using Koha::I18N
First draft for Koha localization using Locale::Maketext
[SIGNED-OFF] First draft for Koha localization using Locale::Maketext
First draft for Koha localization using Locale::Maketext
Translate basket CSV column names using Koha::I18N
[SIGNED_OFF] First draft for Koha localization using Locale::Maketext
Bug 8044 follow-up: Fix warning and error messages in translate.pl
Bug 8044: Translate basket CSV column names using Koha::I18N
Bug 8044 follow-up: Fix warning and error messages in translate.pl
Bug 8044 follow-up: Fix warning and error messages in translate.pl
[SIGNED_OFF] Followup
Followup for pragmas in modules
Bug 8044 follow-up: Add copyright block and use Modern::Perl
[SIGNED-OFF] Bug 8044: First draft for Koha localization using Locale::Maketext
[SIGNED-OFF] Bug 8044 follow-up: Fix warning and error messages in translate.pl
Bug 8044 follow-up: Add copyright block and use Modern::Perl
Bug 8044: Translate basket CSV column names using Koha::I18N
Bug 8044: First draft for Koha localization using Locale::Maketext
Bug 8044 follow-up: Fix warning and error messages in translate.pl
Bug 8044 follow-up: Add copyright block and use Modern::Perl
Bug 8044: Translate basket CSV column names using Koha::I18N
[SIGNED-OFF] Bug 8044: First draft for Koha localization using Locale::Maketext
[SIGNED-OFF] Bug 8044 follow-up: Fix warning and error messages in translate.pl
[SIGNED-OFF] Bug 8044 follow-up: Add copyright block and use Modern::Perl
[SIGNED-OFF] Bug 8044: Translate basket CSV column names using Koha::I18N
Bug 8044: First draft for Koha localization using Locale::Maketext
Bug 8044 follow-up: Fix warning and error messages in translate.pl
Bug 8044 follow-up: Add copyright block and use Modern::Perl
Bug 8044: Translate basket CSV column names using Koha::I18N
Bug 8044: QA Followup to silence koha-qa
Bug 8044: First draft for Koha localization using Locale::Maketext
Bug 8044 follow-up: Fix warning and error messages in translate.pl
Bug 8044 follow-up: Add copyright block and use Modern::Perl
Bug 8044: QA Followup to silence koha-qa

Description Julian Maurice 2012-05-04 12:43:06 UTC
This is a proposal for localization of strings in Perl scripts and modules using Locale::Maketext.
It will allow translating strings without using the templating system.
Note that this is a first draft which could be certainly improved.
Patch and description follows.
Comment 1 Frédéric Demians 2012-05-04 13:00:08 UTC
Have you looked at Locale::TextDomain?
Comment 2 Julian Maurice 2012-05-04 13:05:04 UTC Comment hidden (obsolete)
Comment 3 Julian Maurice 2012-05-04 13:05:42 UTC Comment hidden (obsolete)
Comment 4 Julian Maurice 2012-05-04 13:12:43 UTC
(In reply to comment #1)
> Have you looked at Locale::TextDomain?

Not at all. But, looking at the POD on CPAN, it seems that you cannot set manually the language you want (except using POSIX::setlocale which needs the locale to be installed on the system).
Have you ever used it?
Comment 5 Frédéric Demians 2012-05-04 13:25:10 UTC
> Not at all. But, looking at the POD on CPAN, it seems that you cannot
> set manually the language you want (except using POSIX::setlocale
> which needs the locale to be installed on the system).

I don't know but I'm confident that forcing language selection is
possible with Locale::TextDomain.

> Have you ever used it?

Yes. Please read this:

http://blogs.perl.org/users/aristotle/2011/04/stop-using-maketext.html

Take a look on that:

https://github.com/fredericd/Koha-Contrib-Tamil/blob/master/lib/Koha/Contrib/Tamil/Sitemaper.pm#L64

and the corresponding translation:

https://github.com/fredericd/Koha-Contrib-Tamil/blob/master/po/fr.po#L90
Comment 6 D Ruth Holloway 2012-05-07 11:50:54 UTC
when I try to run update_po.pl, I get a bunch of these:


Use of uninitialized value $_ in concatenation (.) or string at update_po.pl line 23.


followed by:


Extracting strings from source...
/usr/bin/xgettext: no input file given
Try `/usr/bin/xgettext --help' for more information.
system call failed : /usr/bin/xgettext -L Perl --from-code=UTF-8 -kmaketext -o messages.pot -D /home/koha/kohaclone/locale/../. -D /home/koha/kohaclone/locale/../C4 -D /home/koha/kohaclone/locale/../acqui -D /home/koha/kohaclone/locale/../admin -D /home/koha/kohaclone/locale/../authorities -D /home/koha/kohaclone/locale/../basket -D /home/koha/kohaclone/locale/../catalogue -D /home/koha/kohaclone/locale/../cataloguing -D /home/koha/kohaclone/locale/../circ -D /home/koha/kohaclone/locale/../errors -D /home/koha/kohaclone/locale/../Koha -D /home/koha/kohaclone/locale/../labels -D /home/koha/kohaclone/locale/../members -D /home/koha/kohaclone/locale/../misc -D /home/koha/kohaclone/locale/../offline_circ -D /home/koha/kohaclone/locale/../opac -D /home/koha/kohaclone/locale/../patroncards -D /home/koha/kohaclone/locale/../reports -D /home/koha/kohaclone/locale/../reserve -D /home/koha/kohaclone/locale/../reviews -D /home/koha/kohaclone/locale/../rotating_collections -D /home/koha/kohaclone/locale/../serials -D /home/koha/kohaclone/locale/../services -D /home/koha/kohaclone/locale/../sms -D /home/koha/kohaclone/locale/../suggestion -D /home/koha/kohaclone/locale/../tags -D /home/koha/kohaclone/locale/../tools -D /home/koha/kohaclone/locale/../virtualshelves at update_po.pl line 48.


The locale directory I set up is locale/ar_arabic, so I got that part right...what am I missing?
Comment 7 Julian Maurice 2012-05-07 12:19:02 UTC
(In reply to comment #6)
> when I try to run update_po.pl, I get a bunch of these:
> 
> 
> Use of uninitialized value $_ in concatenation (.) or string at update_po.pl
> line 23.

This is the cause of failure. I'm using readdir in a while loop to list all files to pass to xgettext, but apparently this is not supporter by Perl versions prior to 5.11.2 (http://perldoc.perl.org/5.12.0/functions/readdir.html)
I guess you are using Perl 5.10, right?

I will fix that as soon as possible.
Comment 8 Julian Maurice 2012-05-07 13:22:20 UTC Comment hidden (obsolete)
Comment 9 D Ruth Holloway 2012-05-08 07:07:50 UTC Comment hidden (obsolete)
Comment 10 D Ruth Holloway 2012-05-08 07:09:37 UTC
works a treat!  You need to be sure to note that there is a new debian package needed here:

apt-get install liblocale-maketext-lexicon-perl

Thanks for the easy test case, too!
Comment 11 Frédéric Demians 2012-05-08 08:10:25 UTC
I disagree with this patch as it is:

1) Using Locale::Maketext rather than Locale::TextDomain is disputable.
   IMHO Locale::TextDomain manages better parametrized strings and
   plural forms for languages like Russian and Arabic. 
2) Generated .po files are not stored in misc/translator/po, and script
   generating .po files is not in misc/translator. Perl scripts/modules
   translation workflow must be integrated with existing workflow for
   templates and preferences.
3) If we use a new translation framework (Locale::Maketext or
   Locale::TextDomain), it should be a opportunity to reconsider the
   whole Koha translation process. Current process is faulty in many
   places. The work for translators is insanely huge. Strings update
   alway creates a lot of unnecessary 'fuzzy' strings for example. Using
   a translation framework, coupled with Koha coming persistency, it
   would be possible to mark directly in templates strings to translate,
   and then translate them on the fly from .mo files. Advantages: no
   more subdirectories for templates by languages, just .mo files ; no
   need to generate templates for languages anymore.
Comment 12 Julian Maurice 2012-05-10 07:37:11 UTC
Frédéric,

1) First, this patch is not intended to be pushed in master as is, but to start a discussion. So, could you make a counter-proposal patch using Locale::TextDomain?

2) Ok, moving po and script could be done easily.
By 'integrated with existing workflow', do you mean that the po for a language has to be updated when executing './translate update LANG' ?

3) I'm not sure how this can be done, but won't this make the loading of the page slower?
Comment 13 D Ruth Holloway 2012-05-10 07:43:58 UTC
Frédéric,

By the nummbers:

1)  I don't have a firm opinion either way on this, myself, but I do like what I see with Locale::TextDomain; it's pretty nice.  For the sake of readability, I'd put in a vote for Locale::TextDomain.  I'd like to see how we can use it for templates, in another bug, so we don't lose scope here.
2)  That is easily-enough accomplished; I can do that work, if Julian hasn't the time for it; we need *some* solution to this part of the problem right away in the Arabic work I'm on right now.
3)  I concur that we need a larger discussion of the translation mechanism; the vast number of strings that are present in the staff templates, in particular, make translation increasingly hard.  I would like to be part of that discussion, but IMO if we use this as a beachhead for a new way of thinking, then migrate the other parts of the problem into it--where the current mechanism can be used, until we get things switched over fully--then that would probably be best.  So the process becomes:  a) figure out which library to use, and implement for translating strings directly from Perl--the scope of this bug b) move other parts of the current framework into the new mechanism in a way that doesn't slow things down overmuch  c) rejoice!
Comment 14 Julian Maurice 2012-05-11 10:18:19 UTC
I sent an email to koha-devel list to discuss these topics. Setting bug status to 'In discussion'.
Comment 15 Julian Maurice 2012-06-04 12:18:22 UTC
No answers on the mailing list.
I worked on this for better integration into current translation workflow. Patches follow.
Comment 16 Julian Maurice 2012-06-04 12:19:23 UTC Comment hidden (obsolete)
Comment 17 Julian Maurice 2012-06-04 12:20:35 UTC Comment hidden (obsolete)
Comment 18 Julian Maurice 2012-06-04 12:23:56 UTC
The new thing is that locale/ directory and locale/update_po.pl have disappeared. All is now done by misc/translator/translate script.
So you just have to do
  ./translate create LANG
the first time, and then
  ./translate update LANG
every time you want the po to be updated.

PO files are in misc/translator/po/LANG-messages.po
Comment 19 Marcel de Rooy 2012-08-20 08:08:18 UTC Comment hidden (obsolete)
Comment 20 Marcel de Rooy 2012-08-20 08:24:17 UTC
Signing off the patch and reviving the discussion with the following remarks:

The patch adds a dependency: Locale::Maketext,Locale::Maketext::Lexicon. Should be added to PerlDependencies, etc.
I tested the translation with some $lh->maketext call. (Note that the test patch has a conflict.) This works.

When creating a messages-po file, the charset CHARSET is inserted. Can this be prevented? If the user does not correct it, the update process will generate warnings:
Charset "CHARSET" is not a portable encoding name. Message conversion to user's charset might not work.

If there are no messages to translate, you will generate the error: /usr/bin/msgmerge: error while opening "/usr/share/koha/testclone/misc/translator/po/nl-NL-messages.po" for reading: No such file or directory.
This pseudo error can probably be prevented by some simple additional check on existence.

Finally, I did not see a (counter) patch with TextDomain. So I could not look at both sides.
Comment 21 Julian Maurice 2012-08-20 09:46:58 UTC Comment hidden (obsolete)
Comment 22 Julian Maurice 2012-08-20 09:47:50 UTC Comment hidden (obsolete)
Comment 23 Julian Maurice 2012-08-20 09:51:21 UTC
Hi Marcel, and thank you for looking at this :-)

Error messages should not appear anymore with the follow-up.
Conflicts fixed on C4/Acquisition.pm and acqui/basket.pl
Comment 24 Julian Maurice 2012-08-20 10:06:00 UTC Comment hidden (obsolete)
Comment 25 Marcel de Rooy 2012-08-20 10:11:34 UTC
(In reply to comment #24)
> CHARSET is now automatically replaced by UTF-8, and 'update' create the
> PO file if it does not exist.

You added a test in update_messages, but shouldn't you test also in create_messages if there is a file? If there are no messages to translate, the file will not be created. That is not a problem per se; you could just skip the copy statement..
Comment 26 Julian Maurice 2012-08-20 10:25:17 UTC Comment hidden (obsolete)
Comment 27 Marcel de Rooy 2012-08-20 11:37:03 UTC Comment hidden (obsolete)
Comment 28 Marcel de Rooy 2012-08-20 11:38:27 UTC Comment hidden (obsolete)
Comment 29 Marcel de Rooy 2012-08-20 11:39:33 UTC
Signing off the followup from Julian. The second followup speaks for itself.
Comment 30 Marcel de Rooy 2012-08-20 11:40:19 UTC
Julian,
Maybe someone from QA could ask for a copyright and so on.. :)
Comment 31 Julian Maurice 2012-08-20 11:44:09 UTC
Marcel, in "Followup for pragmas in modules" you submitted changes to Koha/Template/Plugin/KohaBranchName.pm. I think this is not wanted.

> Maybe someone from QA could ask for a copyright and so on.. :)
I will add a copyright block, but what do you mean by 'and so on..' ?
Comment 32 Marcel de Rooy 2012-08-20 11:47:00 UTC
(In reply to comment #31)
> Marcel, in "Followup for pragmas in modules" you submitted changes to
> Koha/Template/Plugin/KohaBranchName.pm. I think this is not wanted.
Strictly speaking, you are correct. Just saw those lines..
 
> > Maybe someone from QA could ask for a copyright and so on.. :)
> I will add a copyright block, but what do you mean by 'and so on..' ?
That this was not a full QA but a signoff :) You can leave the status at Signed off IMO BTW.
Comment 33 Julian Maurice 2012-08-20 11:53:18 UTC Comment hidden (obsolete)
Comment 34 Julian Maurice 2012-08-20 11:56:07 UTC
I leave status to Needs Signoff as it still remains one patch not signed-off (attachment 11691 [details] [review]). It's only a test case but it can be integrated to master as well.
Comment 35 Julian Maurice 2012-11-02 13:08:35 UTC Comment hidden (obsolete)
Comment 36 Julian Maurice 2012-11-02 13:08:55 UTC Comment hidden (obsolete)
Comment 37 Julian Maurice 2012-11-02 13:09:11 UTC Comment hidden (obsolete)
Comment 38 Julian Maurice 2012-11-02 13:09:27 UTC Comment hidden (obsolete)
Comment 39 Adrien SAURAT 2013-03-20 14:06:01 UTC
This could be used to translate the SIP files (a lot of messages in English are present in the pm SIP files).
Comment 40 Kyle M Hall 2013-04-26 13:48:49 UTC Comment hidden (obsolete)
Comment 41 Kyle M Hall 2013-04-26 13:49:09 UTC Comment hidden (obsolete)
Comment 42 Kyle M Hall 2013-04-26 13:49:13 UTC Comment hidden (obsolete)
Comment 43 Kyle M Hall 2013-04-26 13:49:17 UTC Comment hidden (obsolete)
Comment 44 Julian Maurice 2013-06-18 09:04:35 UTC Comment hidden (obsolete)
Comment 45 Julian Maurice 2013-06-18 09:04:57 UTC Comment hidden (obsolete)
Comment 46 Julian Maurice 2013-06-18 09:05:25 UTC Comment hidden (obsolete)
Comment 47 Julian Maurice 2013-06-18 09:05:46 UTC Comment hidden (obsolete)
Comment 48 Marcel de Rooy 2013-08-15 13:33:45 UTC Comment hidden (obsolete)
Comment 49 Marcel de Rooy 2013-08-15 13:33:51 UTC Comment hidden (obsolete)
Comment 50 Marcel de Rooy 2013-08-15 13:33:55 UTC Comment hidden (obsolete)
Comment 51 Marcel de Rooy 2013-08-15 13:34:00 UTC Comment hidden (obsolete)
Comment 52 Marcel de Rooy 2013-08-15 13:34:04 UTC Comment hidden (obsolete)
Comment 53 Marcel de Rooy 2013-08-15 19:47:38 UTC
QA Comment:
Final comment. Looks good to me. Older comments have been incorporated.
This has been waiting for long. Thanks for your patience, Julian.

Passed QA
Comment 54 Jonathan Druart 2014-02-21 12:54:41 UTC
(In reply to M. de Rooy from comment #51)
> Created attachment 20369 [details] [review] [review]
> Bug 8044: Translate basket CSV column names using Koha::I18N

Note: this patch is not needed anymore. Bug 10935 fixes the translatable issue on csv headers. But the module and others patches is still needed to translate some others strings (in C4::SIP::* for instance).
Comment 55 Julian Maurice 2014-03-08 10:53:16 UTC
Comment on attachment 20369 [details] [review]
Bug 8044: Translate basket CSV column names using Koha::I18N

> Note: this patch is not needed anymore. Bug 10935 fixes the translatable
> issue on csv headers. But the module and others patches is still needed to
> translate some others strings (in C4::SIP::* for instance).

So I'm marking it as obsolete
Comment 56 Julian Maurice 2014-03-08 10:59:06 UTC
Created attachment 25964 [details] [review]
Bug 8044: First draft for Koha localization using Locale::Maketext

Rebased on master
Comment 57 Julian Maurice 2014-03-08 10:59:33 UTC
Created attachment 25965 [details] [review]
Bug 8044 follow-up: Fix warning and error messages in translate.pl

Rebased on master
Comment 58 Julian Maurice 2014-03-08 10:59:58 UTC
Created attachment 25966 [details] [review]
Bug 8044 follow-up: Add copyright block and use Modern::Perl

Rebased on master
Comment 59 Julian Maurice 2014-03-08 11:00:34 UTC
Created attachment 25967 [details] [review]
Bug 8044: QA Followup to silence koha-qa

Rebased on master
Comment 60 Galen Charlton 2014-05-05 04:56:32 UTC
Pushed to master.  Thanks, Julian!
Comment 61 Fridolin Somers 2014-08-01 13:56:17 UTC
Pushed to 3.14.x, will be in 3.14.10