This is a proposal for localization of strings in Perl scripts and modules using Locale::Maketext. It will allow translating strings without using the templating system. Note that this is a first draft which could be certainly improved. Patch and description follows.
Have you looked at Locale::TextDomain?
Created attachment 9415 [details] [review] First draft for Koha localization using Locale::Maketext You have to use the new module Koha::I18N Code example: use Koha::I18N; use CGI; my $input = new CGI; my $lh = Koha::I18N->get_handle_from_context($input, 'intranet'); print $lh->maketext("Localized string!"); PO files are in locale/LANG/messages.po. To create PO file for a language, just make a directory under locale/ and launch locale/update_po.pl. The directory name must be in lowercase with '_' instead of '-'. For example: for french, create locale/fr_fr/ directory. Missing PO files are created, and existing ones are updated. You can then translate the PO with your favorite editor. Strings will be localized at runtime.
Created attachment 9416 [details] [review] Translate basket CSV column names using Koha::I18N This is a test case for new module Koha::I18N. It translate CSV columns headers when you export a basket as CSV. To test: - create the directory for your language under locale/, for example fr_fr. - run locale/update_po.pl - translate the po locale/fr_fr/messages.po with your favorite po editor. - go to a basket page in staff interface with english interface - click on "Export this basket as CSV" and check that column titles are there and in english - change language to yours and click on the same button. - column names must be translated, according to the PO file.
(In reply to comment #1) > Have you looked at Locale::TextDomain? Not at all. But, looking at the POD on CPAN, it seems that you cannot set manually the language you want (except using POSIX::setlocale which needs the locale to be installed on the system). Have you ever used it?
> Not at all. But, looking at the POD on CPAN, it seems that you cannot > set manually the language you want (except using POSIX::setlocale > which needs the locale to be installed on the system). I don't know but I'm confident that forcing language selection is possible with Locale::TextDomain. > Have you ever used it? Yes. Please read this: http://blogs.perl.org/users/aristotle/2011/04/stop-using-maketext.html Take a look on that: https://github.com/fredericd/Koha-Contrib-Tamil/blob/master/lib/Koha/Contrib/Tamil/Sitemaper.pm#L64 and the corresponding translation: https://github.com/fredericd/Koha-Contrib-Tamil/blob/master/po/fr.po#L90
when I try to run update_po.pl, I get a bunch of these: Use of uninitialized value $_ in concatenation (.) or string at update_po.pl line 23. followed by: Extracting strings from source... /usr/bin/xgettext: no input file given Try `/usr/bin/xgettext --help' for more information. system call failed : /usr/bin/xgettext -L Perl --from-code=UTF-8 -kmaketext -o messages.pot -D /home/koha/kohaclone/locale/../. -D /home/koha/kohaclone/locale/../C4 -D /home/koha/kohaclone/locale/../acqui -D /home/koha/kohaclone/locale/../admin -D /home/koha/kohaclone/locale/../authorities -D /home/koha/kohaclone/locale/../basket -D /home/koha/kohaclone/locale/../catalogue -D /home/koha/kohaclone/locale/../cataloguing -D /home/koha/kohaclone/locale/../circ -D /home/koha/kohaclone/locale/../errors -D /home/koha/kohaclone/locale/../Koha -D /home/koha/kohaclone/locale/../labels -D /home/koha/kohaclone/locale/../members -D /home/koha/kohaclone/locale/../misc -D /home/koha/kohaclone/locale/../offline_circ -D /home/koha/kohaclone/locale/../opac -D /home/koha/kohaclone/locale/../patroncards -D /home/koha/kohaclone/locale/../reports -D /home/koha/kohaclone/locale/../reserve -D /home/koha/kohaclone/locale/../reviews -D /home/koha/kohaclone/locale/../rotating_collections -D /home/koha/kohaclone/locale/../serials -D /home/koha/kohaclone/locale/../services -D /home/koha/kohaclone/locale/../sms -D /home/koha/kohaclone/locale/../suggestion -D /home/koha/kohaclone/locale/../tags -D /home/koha/kohaclone/locale/../tools -D /home/koha/kohaclone/locale/../virtualshelves at update_po.pl line 48. The locale directory I set up is locale/ar_arabic, so I got that part right...what am I missing?
(In reply to comment #6) > when I try to run update_po.pl, I get a bunch of these: > > > Use of uninitialized value $_ in concatenation (.) or string at update_po.pl > line 23. This is the cause of failure. I'm using readdir in a while loop to list all files to pass to xgettext, but apparently this is not supporter by Perl versions prior to 5.11.2 (http://perldoc.perl.org/5.12.0/functions/readdir.html) I guess you are using Perl 5.10, right? I will fix that as soon as possible.
Created attachment 9427 [details] [review] First draft for Koha localization using Locale::Maketext 'while (readdir DIR)' changed to 'foreach (readdir DIR)' to be compatible with Perl 5.10
Created attachment 9449 [details] [SIGNED-OFF] First draft for Koha localization using Locale::Maketext
works a treat! You need to be sure to note that there is a new debian package needed here: apt-get install liblocale-maketext-lexicon-perl Thanks for the easy test case, too!
I disagree with this patch as it is: 1) Using Locale::Maketext rather than Locale::TextDomain is disputable. IMHO Locale::TextDomain manages better parametrized strings and plural forms for languages like Russian and Arabic. 2) Generated .po files are not stored in misc/translator/po, and script generating .po files is not in misc/translator. Perl scripts/modules translation workflow must be integrated with existing workflow for templates and preferences. 3) If we use a new translation framework (Locale::Maketext or Locale::TextDomain), it should be a opportunity to reconsider the whole Koha translation process. Current process is faulty in many places. The work for translators is insanely huge. Strings update alway creates a lot of unnecessary 'fuzzy' strings for example. Using a translation framework, coupled with Koha coming persistency, it would be possible to mark directly in templates strings to translate, and then translate them on the fly from .mo files. Advantages: no more subdirectories for templates by languages, just .mo files ; no need to generate templates for languages anymore.
Frédéric, 1) First, this patch is not intended to be pushed in master as is, but to start a discussion. So, could you make a counter-proposal patch using Locale::TextDomain? 2) Ok, moving po and script could be done easily. By 'integrated with existing workflow', do you mean that the po for a language has to be updated when executing './translate update LANG' ? 3) I'm not sure how this can be done, but won't this make the loading of the page slower?
Frédéric, By the nummbers: 1) I don't have a firm opinion either way on this, myself, but I do like what I see with Locale::TextDomain; it's pretty nice. For the sake of readability, I'd put in a vote for Locale::TextDomain. I'd like to see how we can use it for templates, in another bug, so we don't lose scope here. 2) That is easily-enough accomplished; I can do that work, if Julian hasn't the time for it; we need *some* solution to this part of the problem right away in the Arabic work I'm on right now. 3) I concur that we need a larger discussion of the translation mechanism; the vast number of strings that are present in the staff templates, in particular, make translation increasingly hard. I would like to be part of that discussion, but IMO if we use this as a beachhead for a new way of thinking, then migrate the other parts of the problem into it--where the current mechanism can be used, until we get things switched over fully--then that would probably be best. So the process becomes: a) figure out which library to use, and implement for translating strings directly from Perl--the scope of this bug b) move other parts of the current framework into the new mechanism in a way that doesn't slow things down overmuch c) rejoice!
I sent an email to koha-devel list to discuss these topics. Setting bug status to 'In discussion'.
No answers on the mailing list. I worked on this for better integration into current translation workflow. Patches follow.
Created attachment 9910 [details] [review] First draft for Koha localization using Locale::Maketext You have to use the new module Koha::I18N Code example: use Koha::I18N; use CGI; my $input = new CGI; my $lh = Koha::I18N->get_handle_from_context($input, 'intranet'); print $lh->maketext("Localized string!"); PO files are in misc/translator/po/LANG-messages.po. Creation of PO files are integrated to existing workflow, so to create PO file for a language, just run in misc/translator: ./translate create LANG To update: ./translate update LANG You can then translate the PO with your favorite editor. Strings will be localized at runtime.
Created attachment 9911 [details] [review] Translate basket CSV column names using Koha::I18N This is a test case for new module Koha::I18N. It translate CSV columns headers when you export a basket as CSV. To test: - go to a basket page in staff interface with english interface - click on "Export this basket as CSV" and check that column titles are there and in english - change language and click on the same button. - column titles are still in english - now open a terminal, go to misc/translator and run: ./translate create LANG (e.g. fr-FR) - translate the po misc/translator/po/LANG-messages.po with your favorite po editor. - go back to the navigator and click again on the Export button - column names must be translated, according to the PO file. -- Same patch, but the commit message differ.
The new thing is that locale/ directory and locale/update_po.pl have disappeared. All is now done by misc/translator/translate script. So you just have to do ./translate create LANG the first time, and then ./translate update LANG every time you want the po to be updated. PO files are in misc/translator/po/LANG-messages.po
Created attachment 11689 [details] [review] [SIGNED_OFF] First draft for Koha localization using Locale::Maketext
Signing off the patch and reviving the discussion with the following remarks: The patch adds a dependency: Locale::Maketext,Locale::Maketext::Lexicon. Should be added to PerlDependencies, etc. I tested the translation with some $lh->maketext call. (Note that the test patch has a conflict.) This works. When creating a messages-po file, the charset CHARSET is inserted. Can this be prevented? If the user does not correct it, the update process will generate warnings: Charset "CHARSET" is not a portable encoding name. Message conversion to user's charset might not work. If there are no messages to translate, you will generate the error: /usr/bin/msgmerge: error while opening "/usr/share/koha/testclone/misc/translator/po/nl-NL-messages.po" for reading: No such file or directory. This pseudo error can probably be prevented by some simple additional check on existence. Finally, I did not see a (counter) patch with TextDomain. So I could not look at both sides.
Created attachment 11690 [details] [review] Bug 8044 follow-up: Fix warning and error messages in translate.pl CHARSET is now automatically replaced by UTF-8, and 'update' create the PO file if it does not exist. + add some verbosity
Created attachment 11691 [details] [review] Bug 8044: Translate basket CSV column names using Koha::I18N Rebased on master
Hi Marcel, and thank you for looking at this :-) Error messages should not appear anymore with the follow-up. Conflicts fixed on C4/Acquisition.pm and acqui/basket.pl
Created attachment 11693 [details] [review] Bug 8044 follow-up: Fix warning and error messages in translate.pl CHARSET is now automatically replaced by UTF-8, and 'update' create the PO file if it does not exist. + add some verbosity + add Locale::Maketext and Locale::Maketext::Lexicon to Koha dependencies
(In reply to comment #24) > CHARSET is now automatically replaced by UTF-8, and 'update' create the > PO file if it does not exist. You added a test in update_messages, but shouldn't you test also in create_messages if there is a file? If there are no messages to translate, the file will not be created. That is not a problem per se; you could just skip the copy statement..
Created attachment 11694 [details] [review] Bug 8044 follow-up: Fix warning and error messages in translate.pl CHARSET is now automatically replaced by UTF-8, and 'update' create the PO file if it does not exist. Also do not try to create PO files if POT file creation failed (when there is no messages to translate for example). + add some verbosity + add Locale::Maketext and Locale::Maketext::Lexicon to Koha dependencies
Created attachment 11695 [details] [review] [SIGNED_OFF] Followup
Created attachment 11696 [details] [review] Followup for pragmas in modules
Signing off the followup from Julian. The second followup speaks for itself.
Julian, Maybe someone from QA could ask for a copyright and so on.. :)
Marcel, in "Followup for pragmas in modules" you submitted changes to Koha/Template/Plugin/KohaBranchName.pm. I think this is not wanted. > Maybe someone from QA could ask for a copyright and so on.. :) I will add a copyright block, but what do you mean by 'and so on..' ?
(In reply to comment #31) > Marcel, in "Followup for pragmas in modules" you submitted changes to > Koha/Template/Plugin/KohaBranchName.pm. I think this is not wanted. Strictly speaking, you are correct. Just saw those lines.. > > Maybe someone from QA could ask for a copyright and so on.. :) > I will add a copyright block, but what do you mean by 'and so on..' ? That this was not a full QA but a signoff :) You can leave the status at Signed off IMO BTW.
Created attachment 11697 [details] [review] Bug 8044 follow-up: Add copyright block and use Modern::Perl Obsoletes attachment 11696 [details] [review]: Followup for pragmas in modules
I leave status to Needs Signoff as it still remains one patch not signed-off (attachment 11691 [details] [review]). It's only a test case but it can be integrated to master as well.
Created attachment 13218 [details] [review] [SIGNED-OFF] Bug 8044: First draft for Koha localization using Locale::Maketext Rebased on master
Created attachment 13219 [details] [review] [SIGNED-OFF] Bug 8044 follow-up: Fix warning and error messages in translate.pl Rebased on master
Created attachment 13220 [details] [review] Bug 8044 follow-up: Add copyright block and use Modern::Perl Rebased on master
Created attachment 13221 [details] [review] Bug 8044: Translate basket CSV column names using Koha::I18N Rebased on master
This could be used to translate the SIP files (a lot of messages in English are present in the pm SIP files).
Created attachment 17711 [details] [review] Bug 8044: First draft for Koha localization using Locale::Maketext You have to use the new module Koha::I18N Code example: use Koha::I18N; use CGI; my $input = new CGI; my $lh = Koha::I18N->get_handle_from_context($input, 'intranet'); print $lh->maketext("Localized string!"); PO files are in misc/translator/po/LANG-messages.po. Creation of PO files are integrated to existing workflow, so to create PO file for a language, just run in misc/translator: ./translate create LANG To update: ./translate update LANG You can then translate the PO with your favorite editor. Strings will be localized at runtime. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Works as advertised. Some details needing further attention noted on bug report. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 17712 [details] [review] Bug 8044 follow-up: Fix warning and error messages in translate.pl CHARSET is now automatically replaced by UTF-8, and 'update' create the PO file if it does not exist. Also do not try to create PO files if POT file creation failed (when there is no messages to translate for example). + add some verbosity + add Locale::Maketext and Locale::Maketext::Lexicon to Koha dependencies Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 17713 [details] [review] Bug 8044 follow-up: Add copyright block and use Modern::Perl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 17714 [details] [review] Bug 8044: Translate basket CSV column names using Koha::I18N This is a test case for new module Koha::I18N. It translate CSV columns headers when you export a basket as CSV. To test: - go to a basket page in staff interface with english interface - click on "Export this basket as CSV" and check that column titles are there and in english - change language and click on the same button. - column titles are still in english - now open a terminal, go to misc/translator and run: ./translate create LANG (e.g. fr-FR) - translate the po misc/translator/po/LANG-messages.po with your favorite po editor. - go back to the navigator and click again on the Export button - column names must be translated, according to the PO file. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 19100 [details] [review] [SIGNED-OFF] Bug 8044: First draft for Koha localization using Locale::Maketext Rebased on master
Created attachment 19101 [details] [review] [SIGNED-OFF] Bug 8044 follow-up: Fix warning and error messages in translate.pl Rebased on master
Created attachment 19102 [details] [review] [SIGNED-OFF] Bug 8044 follow-up: Add copyright block and use Modern::Perl Rebased on master
Created attachment 19103 [details] [review] [SIGNED-OFF] Bug 8044: Translate basket CSV column names using Koha::I18N Rebased on master
Created attachment 20366 [details] [review] Bug 8044: First draft for Koha localization using Locale::Maketext You have to use the new module Koha::I18N Code example: use Koha::I18N; use CGI; my $input = new CGI; my $lh = Koha::I18N->get_handle_from_context($input, 'intranet'); print $lh->maketext("Localized string!"); PO files are in misc/translator/po/LANG-messages.po. Creation of PO files are integrated to existing workflow, so to create PO file for a language, just run in misc/translator: ./translate create LANG To update: ./translate update LANG You can then translate the PO with your favorite editor. Strings will be localized at runtime. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Works as advertised. Some details needing further attention noted on bug report. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 20367 [details] [review] Bug 8044 follow-up: Fix warning and error messages in translate.pl CHARSET is now automatically replaced by UTF-8, and 'update' create the PO file if it does not exist. Also do not try to create PO files if POT file creation failed (when there is no messages to translate for example). + add some verbosity + add Locale::Maketext and Locale::Maketext::Lexicon to Koha dependencies Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Created attachment 20368 [details] [review] Bug 8044 follow-up: Add copyright block and use Modern::Perl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 20369 [details] [review] Bug 8044: Translate basket CSV column names using Koha::I18N This is a test case for new module Koha::I18N. It translate CSV columns headers when you export a basket as CSV. To test: - go to a basket page in staff interface with english interface - click on "Export this basket as CSV" and check that column titles are there and in english - change language and click on the same button. - column titles are still in english - now open a terminal, go to misc/translator and run: ./translate create LANG (e.g. fr-FR) - translate the po misc/translator/po/LANG-messages.po with your favorite po editor. - go back to the navigator and click again on the Export button - column names must be translated, according to the PO file. Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 20370 [details] [review] Bug 8044: QA Followup to silence koha-qa Just moving the line for Modern::Perl Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
QA Comment: Final comment. Looks good to me. Older comments have been incorporated. This has been waiting for long. Thanks for your patience, Julian. Passed QA
(In reply to M. de Rooy from comment #51) > Created attachment 20369 [details] [review] [review] > Bug 8044: Translate basket CSV column names using Koha::I18N Note: this patch is not needed anymore. Bug 10935 fixes the translatable issue on csv headers. But the module and others patches is still needed to translate some others strings (in C4::SIP::* for instance).
Comment on attachment 20369 [details] [review] Bug 8044: Translate basket CSV column names using Koha::I18N > Note: this patch is not needed anymore. Bug 10935 fixes the translatable > issue on csv headers. But the module and others patches is still needed to > translate some others strings (in C4::SIP::* for instance). So I'm marking it as obsolete
Created attachment 25964 [details] [review] Bug 8044: First draft for Koha localization using Locale::Maketext Rebased on master
Created attachment 25965 [details] [review] Bug 8044 follow-up: Fix warning and error messages in translate.pl Rebased on master
Created attachment 25966 [details] [review] Bug 8044 follow-up: Add copyright block and use Modern::Perl Rebased on master
Created attachment 25967 [details] [review] Bug 8044: QA Followup to silence koha-qa Rebased on master
Pushed to master. Thanks, Julian!
Pushed to 3.14.x, will be in 3.14.10