From 4738b3c6213bf1ebfbc0f8be7b42f87334923758 Mon Sep 17 00:00:00 2001 From: Julian Maurice Date: Sun, 29 Mar 2020 14:56:20 +0200 Subject: [PATCH] Bug 25067: Move PO file manipulation code into gulp tasks misc/translator/translate was doing three different things: - extract translatable strings - create or update PO files - install translated templates This patch separates responsibilities by moving the string extraction code into several 'xgettext-like' scripts and adds gulp tasks to automate string extraction and PO files update This has several benefits: - gulp runs tasks in parallel, so it's a lot faster (updating all PO files is at least 10 times faster with my 4-cores CPU) - there is no need for $KOHA_CONF to be defined LangInstaller.pm relied on $KOHA_CONF to get the different paths needed. I'm not sure why, since string extraction and PO update should work on source files, not installed files - string extraction code can be more easily tested This patch also brings a couple of fixes and improvements: - TT string extraction (strings wrapped in [% t(...) %]) was done with Template::Parser and PPI, which was extremely slow, and had some problems (see bug 24797). This is now done with Locale::XGettext::TT2 (new dependency) which is a lot faster, and fixes bug 24797 - string extraction for system preferences was putting context inside msgid. For instance: # Accounting > Policy msgid "accounting.pref#FinePaymentAutoPopup# automatically display " "a print dialog for a payment receipt when making a payment.." Now context is put into msgctxt, and the reference is set, which is cleaner #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref msgctxt "Accounting > Policy > FinePaymentAutoPopup" msgid "automatically display a print dialog for a payment receipt " "when making a payment.." The downside is that some messages will have to be re-translated, especially short messages like 'Do', for which msgmerge has a hard time finding the corresponding new msgid. - Fix header in 4 PO files Test plan: 1. Run `yarn install` and install Locale::XGettext::TT2 2. Run `gulp po:update` 3. Verify the contents of updated PO files 4. Run `cd misc/translator && ./translate install ` 5. Verify that all (templates, sysprefs, xslt, installer files) is correctly translated 6. Run `gulp po:create --lang ` and verify that it created all PO files for that language 7. Run `prove t/misc/translator` Signed-off-by: Bernardo Gonzalez Kriegel Need to install yarn & gulp, no errors --- cpanfile | 2 +- docs/development/internationalization.md | 121 +++ gulpfile.js | 293 ++++++- misc/translator/LangInstaller.pm | 781 ++---------------- misc/translator/po/dz-pref.po | 10 +- misc/translator/po/gd-pref.po | 10 +- misc/translator/po/lv-pref.po | 10 +- misc/translator/po/te-pref.po | 10 +- misc/translator/tmpl_process3.pl | 122 +-- misc/translator/translate | 42 +- misc/translator/xgettext-installer | 143 ++++ misc/translator/xgettext-pref | 134 +++ misc/translator/xgettext-tt2 | 41 + misc/translator/xgettext.pl | 2 +- package.json | 3 + t/LangInstaller.t | 109 --- t/misc/translator/sample.pref | 14 + .../simple.tt => misc/translator/sample.tt} | 2 +- t/misc/translator/sample.yml | 15 + t/misc/translator/xgettext-installer.t | 32 + t/misc/translator/xgettext-pref.t | 61 ++ t/misc/translator/xgettext-tt2.t | 74 ++ yarn.lock | 112 ++- 23 files changed, 1169 insertions(+), 974 deletions(-) create mode 100644 docs/development/internationalization.md create mode 100755 misc/translator/xgettext-installer create mode 100755 misc/translator/xgettext-pref create mode 100755 misc/translator/xgettext-tt2 delete mode 100644 t/LangInstaller.t create mode 100644 t/misc/translator/sample.pref rename t/{LangInstaller/templates/simple.tt => misc/translator/sample.tt} (96%) create mode 100644 t/misc/translator/sample.yml create mode 100644 t/misc/translator/xgettext-installer.t create mode 100644 t/misc/translator/xgettext-pref.t create mode 100755 t/misc/translator/xgettext-tt2.t diff --git a/cpanfile b/cpanfile index 7e5eeaa1d8..9f5479a141 100644 --- a/cpanfile +++ b/cpanfile @@ -143,6 +143,7 @@ recommends 'Gravatar::URL', '1.03'; recommends 'HTTPD::Bench::ApacheBench', '0.73'; recommends 'LWP::Protocol::https', '5.836'; recommends 'Lingua::Ispell', '0.07'; +recommends 'Locale::XGettext::TT2', '0.7'; recommends 'Module::Bundled::Files', '0.03'; recommends 'Module::Load::Conditional', '0.38'; recommends 'Module::Pluggable', '3.9'; @@ -154,7 +155,6 @@ recommends 'Net::SFTP::Foreign', '1.73'; recommends 'Net::Server', '0.97'; recommends 'Net::Z3950::SimpleServer', '1.15'; recommends 'PDF::FromHTML', '0.31'; -recommends 'PPI', '1.215'; recommends 'Parallel::ForkManager', '0.75'; recommends 'Readonly', '0.01'; recommends 'Readonly::XS', '0.01'; diff --git a/docs/development/internationalization.md b/docs/development/internationalization.md new file mode 100644 index 0000000000..0365fefb9b --- /dev/null +++ b/docs/development/internationalization.md @@ -0,0 +1,121 @@ +# Internationalization + +This page documents how internationalization works in Koha. + +## Making strings translatable + +There are several ways of making a string translatable, depending on where it +is located + +### In Template::Toolkit files (`*.tt`) + +The simplest way to make a string translatable in a template is to do nothing. +Templates are parsed as HTML files and almost all text nodes are considered as +translatable strings. This also includes some attributes like `title` and +`placeholder`. + +This method has some downsides: you don't have full control over what would +appear in PO files and you cannot use plural forms or context. In order to do +that you have to use `i18n.inc` + +`i18n.inc` contains several macros that, when used, make a string translatable. +The first thing to do is to make these macros available by adding + + [% PROCESS 'i18n.inc' %] + +at the top of the template file. Then you can use those macros. + +The simplest one is `t(msgid)` + + [% t('This is a translatable string') %] + +You can also use variable substitution with `tx(msgid, vars)` + + [% tx('Hello, {name}', { name = 'World' }) %] + +You can use plural forms with `tn(msgid, msgid_plural, count)` + + [% tn('a child', 'several children', number_of_children) %] + +You can add context, to help translators when a term is ambiguous, with +`tp(msgctxt, msgid)` + + [% tp('verb', 'order') %] + [% tp('noun', 'order') %] + +Or any combinations of the above + + [% tnpx('bibliographic record', '{count} item', '{count} items', items_count, { count = items_count }) %] + +### In JavaScript files (`*.js`) + +Like in templates, you have several functions available. Just replace `t` by `__`. + + __('This is a translatable string'); + __npx('bibliographic record, '{count} item', '{count} items', items_count, { count: items_count }); + +### In Perl files (`*.pl`, `*.pm`) + +You will have to add + + use Koha::I18N; + +at the top of the file, and then the same functions as above will be available. + + __('This is a translatable string'); + __npx('bibliographic record, '{count} item', '{count} items', $items_count, count => $items_count); + +### In installer and preferences YAML files (`*.yml`) + +Nothing special to do here. All strings will be automatically translatable. + +## Manipulating PO files + +Once strings have been made translatable in source files, they have to be +extracted into PO files and uploaded on https://translate.koha-community.org/ +so they can be translated. + +### Install gulp first + +The next sections rely on gulp. If it's not installed, run the following +commands: + + # as root + npm install gulp-cli -g + + # as normal user, from the root of Koha repository + yarn + +### Create PO files for a new language + +If you want to add translations for a new language, you have to create the +missing PO files. You can do that by executing the following command: + + # Replace xx-XX by your language tag + gulp po:create --lang xx-XX + +New PO files will be available in `misc/translator/po`. + +### Update PO files with new strings + +When new features or bugfixes are added to Koha, new translatable strings can +be added, other can be removed or modified, and the PO file become out of sync. + +To be able to translate the new or modified strings, you have to update PO +files. This can be done by executing the following command: + + # Update PO files for all languages + gulp po:update + + # or only one language + gulp po:update --lang xx-XX + +### Only extract strings + +Creating or updating PO files automatically extract strings, but if for some +reasons you want to only extract strings without touching PO files, you can run +the following command: + + gulp po:extract + +POT files will be available in `misc/translator`. diff --git a/gulpfile.js b/gulpfile.js index 5b8d6e68a2..f2ac6298dc 100644 --- a/gulpfile.js +++ b/gulpfile.js @@ -1,12 +1,23 @@ /* eslint-env node */ /* eslint no-console:"off" */ -const { dest, series, src, watch } = require('gulp'); +const { dest, parallel, series, src, watch } = require('gulp'); + +const child_process = require('child_process'); +const fs = require('fs'); +const os = require('os'); +const path = require('path'); +const util = require('util'); const sass = require("gulp-sass"); const cssnano = require("gulp-cssnano"); const sourcemaps = require('gulp-sourcemaps'); const autoprefixer = require('gulp-autoprefixer'); +const concatPo = require('gulp-concat-po'); +const exec = require('gulp-exec'); +const merge = require('merge-stream'); +const through2 = require('through2'); +const Vinyl = require('vinyl'); const args = require('minimist')(process.argv.slice(2)); const STAFF_JS_BASE = "koha-tmpl/intranet-tmpl/prog/js"; @@ -46,8 +57,288 @@ function build() { .pipe(dest(css_base)); } +const poTasks = { + 'marc-MARC21': { + extract: po_extract_marc_marc21, + create: po_create_marc_marc21, + update: po_update_marc_marc21, + }, + 'marc-NORMARC': { + extract: po_extract_marc_normarc, + create: po_create_marc_normarc, + update: po_update_marc_normarc, + }, + 'marc-UNIMARC': { + extract: po_extract_marc_unimarc, + create: po_create_marc_unimarc, + update: po_update_marc_unimarc, + }, + 'staff-prog': { + extract: po_extract_staff, + create: po_create_staff, + update: po_update_staff, + }, + 'opac-bootstrap': { + extract: po_extract_opac, + create: po_create_opac, + update: po_update_opac, + }, + 'pref': { + extract: po_extract_pref, + create: po_create_pref, + update: po_update_pref, + }, + 'messages': { + extract: po_extract_messages, + create: po_create_messages, + update: po_update_messages, + }, + 'messages-js': { + extract: po_extract_messages_js, + create: po_create_messages_js, + update: po_update_messages_js, + }, + 'installer': { + extract: po_extract_installer, + create: po_create_installer, + update: po_update_installer, + }, + 'installer-MARC21': { + extract: po_extract_installer_marc21, + create: po_create_installer_marc21, + update: po_update_installer_marc21, + }, +}; + +const poTypes = Object.keys(poTasks); + +function po_extract_marc (type) { + return src(`koha-tmpl/*-tmpl/*/en/**/*${type}*`, { read: false, nocase: true }) + .pipe(xgettext('misc/translator/xgettext.pl --charset=UTF-8 -s', `Koha-marc-${type}.pot`)) + .pipe(dest('misc/translator')) +} + +function po_extract_marc_marc21 () { return po_extract_marc('MARC21') } +function po_extract_marc_normarc () { return po_extract_marc('NORMARC') } +function po_extract_marc_unimarc () { return po_extract_marc('UNIMARC') } + +function po_extract_staff () { + const globs = [ + 'koha-tmpl/intranet-tmpl/prog/en/**/*.tt', + 'koha-tmpl/intranet-tmpl/prog/en/**/*.inc', + '!koha-tmpl/intranet-tmpl/prog/en/**/*MARC21*', + '!koha-tmpl/intranet-tmpl/prog/en/**/*NORMARC*', + '!koha-tmpl/intranet-tmpl/prog/en/**/*UNIMARC*', + ]; + + return src(globs, { read: false, nocase: true }) + .pipe(xgettext('misc/translator/xgettext.pl --charset=UTF-8 -s', 'Koha-staff-prog.pot')) + .pipe(dest('misc/translator')) +} + +function po_extract_opac () { + const globs = [ + 'koha-tmpl/opac-tmpl/bootstrap/en/**/*.tt', + 'koha-tmpl/opac-tmpl/bootstrap/en/**/*.inc', + '!koha-tmpl/opac-tmpl/bootstrap/en/**/*MARC21*', + '!koha-tmpl/opac-tmpl/bootstrap/en/**/*NORMARC*', + '!koha-tmpl/opac-tmpl/bootstrap/en/**/*UNIMARC*', + ]; + + return src(globs, { read: false, nocase: true }) + .pipe(xgettext('misc/translator/xgettext.pl --charset=UTF-8 -s', 'Koha-opac-bootstrap.pot')) + .pipe(dest('misc/translator')) +} + +const xgettext_options = '--from-code=UTF-8 --package-name Koha ' + + '--package-version= -k -k__ -k__x -k__n:1,2 -k__nx:1,2 -k__xn:1,2 ' + + '-k__p:1c,2 -k__px:1c,2 -k__np:1c,2,3 -k__npx:1c,2,3 -kN__ ' + + '-kN__n:1,2 -kN__p:1c,2 -kN__np:1c,2,3 --force-po'; + +function po_extract_messages_js () { + const globs = [ + 'koha-tmpl/intranet-tmpl/prog/js/**/*.js', + 'koha-tmpl/opac-tmpl/bootstrap/js/**/*.js', + ]; + + return src(globs, { read: false, nocase: true }) + .pipe(xgettext(`xgettext -L JavaScript ${xgettext_options}`, 'Koha-messages-js.pot')) + .pipe(dest('misc/translator')) +} + +function po_extract_messages () { + const perlStream = src(['**/*.pl', '**/*.pm'], { read: false, nocase: true }) + .pipe(xgettext(`xgettext -L Perl ${xgettext_options}`, 'Koha-perl.pot')) + + const ttStream = src([ + 'koha-tmpl/intranet-tmpl/prog/en/**/*.tt', + 'koha-tmpl/intranet-tmpl/prog/en/**/*.inc', + 'koha-tmpl/opac-tmpl/bootstrap/en/**/*.tt', + 'koha-tmpl/opac-tmpl/bootstrap/en/**/*.inc', + ], { read: false, nocase: true }) + .pipe(xgettext('misc/translator/xgettext-tt2 --from-code=UTF-8', 'Koha-tt.pot')) + + const headers = { + 'Project-Id-Version': 'Koha', + 'Content-Type': 'text/plain; charset=UTF-8', + }; + + return merge(perlStream, ttStream) + .pipe(concatPo('Koha-messages.pot', { headers })) + .pipe(dest('misc/translator')) +} + +function po_extract_pref () { + return src('koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/*.pref', { read: false }) + .pipe(xgettext('misc/translator/xgettext-pref', 'Koha-pref.pot')) + .pipe(dest('misc/translator')) +} + +function po_extract_installer () { + const globs = [ + 'installer/data/mysql/en/mandatory/*.yml', + 'installer/data/mysql/en/optional/*.yml', + ]; + + return src(globs, { read: false, nocase: true }) + .pipe(xgettext('misc/translator/xgettext-installer', 'Koha-installer.pot')) + .pipe(dest('misc/translator')) +} + +function po_extract_installer_marc (type) { + const globs = `installer/data/mysql/en/marcflavour/${type}/**/*.yml`; + + return src(globs, { read: false, nocase: true }) + .pipe(xgettext('misc/translator/xgettext-installer', `Koha-installer-${type}.pot`)) + .pipe(dest('misc/translator')) +} + +function po_extract_installer_marc21 () { return po_extract_installer_marc('MARC21') } + +function po_create_type (type) { + const access = util.promisify(fs.access); + const exec = util.promisify(child_process.exec); + + const languages = getLanguages(); + const promises = []; + for (const language of languages) { + const locale = language.split('-').filter(s => s.length !== 4).join('_'); + const po = `misc/translator/po/${language}-${type}.po`; + const pot = `misc/translator/Koha-${type}.pot`; + + const promise = access(po) + .catch(() => exec(`msginit -o ${po} -i ${pot} -l ${locale} --no-translator`)) + promises.push(promise); + } + + return Promise.all(promises); +} + +function po_create_marc_marc21 () { return po_create_type('marc-MARC21') } +function po_create_marc_normarc () { return po_create_type('marc-NORMARC') } +function po_create_marc_unimarc () { return po_create_type('marc-UNIMARC') } +function po_create_staff () { return po_create_type('staff-prog') } +function po_create_opac () { return po_create_type('opac-bootstrap') } +function po_create_pref () { return po_create_type('pref') } +function po_create_messages () { return po_create_type('messages') } +function po_create_messages_js () { return po_create_type('messages-js') } +function po_create_installer () { return po_create_type('installer') } +function po_create_installer_marc21 () { return po_create_type('installer-MARC21') } + +function po_update_type (type) { + const msgmerge_opts = '--backup=off --quiet --sort-output --update'; + const cmd = `msgmerge ${msgmerge_opts} <%= file.path %> misc/translator/Koha-${type}.pot`; + const languages = getLanguages(); + const globs = languages.map(language => `misc/translator/po/${language}-${type}.po`); + + return src(globs) + .pipe(exec(cmd, { continueOnError: true })) + .pipe(exec.reporter({ err: false, stdout: false })) +} + +function po_update_marc_marc21 () { return po_update_type('marc-MARC21') } +function po_update_marc_normarc () { return po_update_type('marc-NORMARC') } +function po_update_marc_unimarc () { return po_update_type('marc-UNIMARC') } +function po_update_staff () { return po_update_type('staff-prog') } +function po_update_opac () { return po_update_type('opac-bootstrap') } +function po_update_pref () { return po_update_type('pref') } +function po_update_messages () { return po_update_type('messages') } +function po_update_messages_js () { return po_update_type('messages-js') } +function po_update_installer () { return po_update_type('installer') } +function po_update_installer_marc21 () { return po_update_type('installer-MARC21') } + +/** + * Gulp plugin that executes xgettext-like command `cmd` on all files given as + * input, and then outputs the result as a POT file named `filename`. + * `cmd` should accept -o and -f options + */ +function xgettext (cmd, filename) { + const filenames = []; + + function transform (file, encoding, callback) { + filenames.push(path.relative(file.cwd, file.path)); + callback(); + } + + function flush (callback) { + fs.mkdtemp(path.join(os.tmpdir(), 'koha-'), (err, folder) => { + const outputFilename = path.join(folder, filename); + const filesFilename = path.join(folder, 'files'); + fs.writeFile(filesFilename, filenames.join(os.EOL), err => { + if (err) return callback(err); + + const command = `${cmd} -o ${outputFilename} -f ${filesFilename}`; + child_process.exec(command, err => { + if (err) return callback(err); + + fs.readFile(outputFilename, (err, data) => { + if (err) return callback(err); + + const file = new Vinyl(); + file.path = path.join(file.base, filename); + file.contents = data; + callback(null, file); + }); + }); + }); + }) + } + + return through2.obj(transform, flush); +} + +/** + * Return languages selected for PO-related tasks + * + * This can be either languages given on command-line with --lang option, or + * all the languages found in misc/translator/po otherwise + */ +function getLanguages () { + if (Array.isArray(args.lang)) { + return args.lang; + } + + if (args.lang) { + return [args.lang]; + } + + const filenames = fs.readdirSync('misc/translator/po') + .filter(filename => filename.endsWith('.po')) + .filter(filename => !filename.startsWith('.')) + + const re = new RegExp('-(' + poTypes.join('|') + ')\.po$'); + languages = filenames.map(filename => filename.replace(re, '')) + + return Array.from(new Set(languages)); +} + exports.build = build; exports.css = css; + +exports['po:create'] = parallel(...poTypes.map(type => series(poTasks[type].extract, poTasks[type].create))); +exports['po:update'] = parallel(...poTypes.map(type => series(poTasks[type].extract, poTasks[type].update))); +exports['po:extract'] = parallel(...poTypes.map(type => poTasks[type].extract)); + exports.default = function () { watch(css_base + "/src/**/*.scss", series('css')); } diff --git a/misc/translator/LangInstaller.pm b/misc/translator/LangInstaller.pm index b9c5c0d161..be51394452 100644 --- a/misc/translator/LangInstaller.pm +++ b/misc/translator/LangInstaller.pm @@ -22,36 +22,15 @@ use Modern::Perl; use C4::Context; # WARNING: Any other tested YAML library fails to work properly in this # script content -use YAML::Syck qw( Dump LoadFile DumpFile ); +use YAML::Syck qw( LoadFile DumpFile ); use Locale::PO; use FindBin qw( $Bin ); use File::Basename; -use File::Find; use File::Path qw( make_path ); use File::Copy; -use File::Slurp; -use File::Spec; -use File::Temp qw( tempdir tempfile ); -use Template::Parser; -use PPI; - $YAML::Syck::ImplicitTyping = 1; - -# Default file header for .po syspref files -my $default_pref_po_header = Locale::PO->new(-msgid => '', -msgstr => - "Project-Id-Version: PACKAGE VERSION\\n" . - "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\n" . - "Last-Translator: FULL NAME \\n" . - "Language-Team: Koha Translate List \\n" . - "MIME-Version: 1.0\\n" . - "Content-Type: text/plain; charset=UTF-8\\n" . - "Content-Transfer-Encoding: 8bit\\n" . - "Plural-Forms: nplurals=2; plural=(n > 1);\\n" -); - - sub set_lang { my ($self, $lang) = @_; @@ -60,7 +39,6 @@ sub set_lang { "/prog/$lang/modules/admin/preferences"; } - sub new { my ($class, $lang, $pref_only, $verbose) = @_; @@ -75,32 +53,16 @@ sub new { $self->{verbose} = $verbose; $self->{process} = "$Bin/tmpl_process3.pl " . ($verbose ? '' : '-q'); $self->{path_po} = "$Bin/po"; - $self->{po} = { '' => $default_pref_po_header }; + $self->{po} = {}; $self->{domain} = 'Koha'; - $self->{cp} = `which cp`; - $self->{msgmerge} = `which msgmerge`; $self->{msgfmt} = `which msgfmt`; - $self->{msginit} = `which msginit`; - $self->{msgattrib} = `which msgattrib`; - $self->{xgettext} = `which xgettext`; - $self->{sed} = `which sed`; $self->{po2json} = "$Bin/po2json"; $self->{gzip} = `which gzip`; $self->{gunzip} = `which gunzip`; - chomp $self->{cp}; - chomp $self->{msgmerge}; chomp $self->{msgfmt}; - chomp $self->{msginit}; - chomp $self->{msgattrib}; - chomp $self->{xgettext}; - chomp $self->{sed}; chomp $self->{gzip}; chomp $self->{gunzip}; - unless ($self->{xgettext}) { - die "Missing 'xgettext' executable. Have you installed the gettext package?\n"; - } - # Get all .pref file names opendir my $fh, $self->{path_pref_en}; my @pref_files = grep { /\.pref$/ } readdir($fh); @@ -175,7 +137,6 @@ sub new { bless $self, $class; } - sub po_filename { my $self = shift; my $suffix = shift; @@ -186,162 +147,90 @@ sub po_filename { return $trans_file; } +sub get_trans_text { + my ($self, $msgctxt, $msgid) = @_; -sub po_append { - my ($self, $id, $comment) = @_; - my $po = $self->{po}; - my $p = $po->{$id}; - if ( $p ) { - $p->comment( $p->comment . "\n" . $comment ); - } - else { - $po->{$id} = Locale::PO->new( - -comment => $comment, - -msgid => $id, - -msgstr => '' - ); - } -} - + my $key = ($msgctxt // '') . ";$msgid"; -sub add_prefs { - my ($self, $comment, $prefs) = @_; + my $po = $self->{po}->{$key}; + if ($po) { + my $msgstr = Locale::PO->dequote($po->msgstr); - for my $pref ( @$prefs ) { - my $pref_name = ''; - for my $element ( @$pref ) { - if ( ref( $element) eq 'HASH' ) { - $pref_name = $element->{pref}; - last; - } - } - for my $element ( @$pref ) { - if ( ref( $element) eq 'HASH' ) { - while ( my ($key, $value) = each(%$element) ) { - next unless $key eq 'choices' or $key eq 'multiple'; - next unless ref($value) eq 'HASH'; - for my $ckey ( keys %$value ) { - my $id = $self->{file} . "#$pref_name# " . $value->{$ckey}; - $self->po_append( $id, $comment ); - } - } - } - elsif ( $element ) { - $self->po_append( $self->{file} . "#$pref_name# $element", $comment ); - } - } + return $msgstr || $msgid; } -} - -sub get_trans_text { - my ($self, $id) = @_; - - my $po = $self->{po}->{$id}; - return unless $po; - return Locale::PO->dequote($po->msgstr); + return $msgid; } +sub get_translated_tab_content { + my ($self, $tab, $tab_content) = @_; -sub update_tab_prefs { - my ($self, $pref, $prefs) = @_; - - for my $p ( @$prefs ) { - my $pref_name = ''; - next unless $p; - for my $element ( @$p ) { - if ( ref( $element) eq 'HASH' ) { - $pref_name = $element->{pref}; - last; - } - } - for my $i ( 0..@$p-1 ) { - my $element = $p->[$i]; - if ( ref( $element) eq 'HASH' ) { - while ( my ($key, $value) = each(%$element) ) { - next unless $key eq 'choices' or $key eq 'multiple'; - next unless ref($value) eq 'HASH'; - for my $ckey ( keys %$value ) { - my $id = $self->{file} . "#$pref_name# " . $value->{$ckey}; - my $text = $self->get_trans_text( $id ); - $value->{$ckey} = $text if $text; - } - } - } - elsif ( $element ) { - my $id = $self->{file} . "#$pref_name# $element"; - my $text = $self->get_trans_text( $id ); - $p->[$i] = $text if $text; - } - } + if ( ref($tab_content) eq 'ARRAY' ) { + return $self->get_translated_prefs($tab, $tab_content); } -} + my $translated_tab_content = { + map { + my $section = $_; + my $sysprefs = $tab_content->{$section}; + my $context = "$tab > $section"; -sub get_po_from_prefs { - my $self = shift; + $self->get_trans_text($tab, $section) => $self->get_translated_prefs($context, $sysprefs); + } keys %$tab_content + }; - for my $file ( @{$self->{pref_files}} ) { - my $pref = LoadFile( $self->{path_pref_en} . "/$file" ); - $self->{file} = $file; - # Entries for tab titles - $self->po_append( $self->{file}, $_ ) for keys %$pref; - while ( my ($tab, $tab_content) = each %$pref ) { - if ( ref($tab_content) eq 'ARRAY' ) { - $self->add_prefs( $tab, $tab_content ); - next; - } - while ( my ($section, $sysprefs) = each %$tab_content ) { - my $comment = "$tab > $section"; - $self->po_append( $self->{file} . " " . $section, $comment ); - $self->add_prefs( $comment, $sysprefs ); - } - } - } + return $translated_tab_content; } +sub get_translated_prefs { + my ($self, $context, $sysprefs) = @_; -sub save_po { - my $self = shift; + my $translated_prefs = [ + map { + my ($pref_elt) = grep { ref($_) eq 'HASH' && exists $_->{pref} } @$_; + my $pref_name = $pref_elt ? $pref_elt->{pref} : ''; - # Create file header if it doesn't already exist - my $po = $self->{po}; - $po->{''} ||= $default_pref_po_header; + my $translated_syspref = [ + map { + $self->get_translated_pref("$context > $pref_name", $_ ); + } @$_ + ]; - # Write .po entries into a file put in Koha standard po directory - Locale::PO->save_file_fromhash( $self->po_filename("-pref.po"), $po ); - say "Saved in file: ", $self->po_filename("-pref.po") if $self->{verbose}; -} + $translated_syspref; + } @$sysprefs + ]; + return $translated_prefs; +} -sub get_po_merged_with_en { - my $self = shift; +sub get_translated_pref { + my ($self, $context, $syspref) = @_; - # Get po from current 'en' .pref files - $self->get_po_from_prefs(); - my $po_current = $self->{po}; + unless (ref($syspref)) { + return $self->get_trans_text($context, $syspref // ''); + } - # Get po from previous generation - my $po_previous = Locale::PO->load_file_ashash( $self->po_filename("-pref.po") ); + my $translated_pref = { + map { + my $key = $_; + my $value = $syspref->{$key}; - for my $id ( keys %$po_current ) { - my $po = $po_previous->{Locale::PO->quote($id)}; - next unless $po; - my $text = Locale::PO->dequote( $po->msgstr ); - $po_current->{$id}->msgstr( $text ); - } -} + my $translated_value = $value; + if (($key eq 'choices' || $key eq 'multiple') && ref($value) eq 'HASH') { + $translated_value = { + map { + $_ => $self->get_trans_text($context, $value->{$_}) + } keys %$value + } + } + $key => $translated_value + } keys %$syspref + }; -sub update_prefs { - my $self = shift; - print "Update '", $self->{lang}, - "' preferences .po file from 'en' .pref files\n" if $self->{verbose}; - $self->get_po_merged_with_en(); - $self->save_po(); + return $translated_pref; } - sub install_prefs { my $self = shift; @@ -350,45 +239,30 @@ sub install_prefs { exit; } - # Get the language .po file merged with last modified 'en' preferences - $self->get_po_merged_with_en(); + my @po_entries = @{ Locale::PO->load_file_asarray($self->po_filename("-pref.po"), 'utf8') }; + $self->{po} = { map { + my $msgctxt = $_->msgctxt ? Locale::PO->dequote($_->msgctxt) : ''; + my $msgid = Locale::PO->dequote($_->msgid); + + "$msgctxt;$msgid" => $_; + } @po_entries }; for my $file ( @{$self->{pref_files}} ) { my $pref = LoadFile( $self->{path_pref_en} . "/$file" ); - $self->{file} = $file; - # First, keys are replaced (tab titles) - $pref = do { - my %pref = map { - $self->get_trans_text( $self->{file} ) || $_ => $pref->{$_} - } keys %$pref; - \%pref; + + my $translated_pref = { + map { + my $tab = $_; + my $tab_content = $pref->{$tab}; + + $self->get_trans_text(undef, $tab) => $self->get_translated_tab_content($tab, $tab_content); + } keys %$pref }; - while ( my ($tab, $tab_content) = each %$pref ) { - if ( ref($tab_content) eq 'ARRAY' ) { - $self->update_tab_prefs( $pref, $tab_content ); - next; - } - while ( my ($section, $sysprefs) = each %$tab_content ) { - $self->update_tab_prefs( $pref, $sysprefs ); - } - my $ntab = {}; - for my $section ( keys %$tab_content ) { - my $id = $self->{file} . " $section"; - my $text = $self->get_trans_text($id); - my $nsection = $text ? $text : $section; - if( exists $ntab->{$nsection} ) { - # When translations collide (see BZ 18634) - push @{$ntab->{$nsection}}, @{$tab_content->{$section}}; - } else { - $ntab->{$nsection} = $tab_content->{$section}; - } - } - $pref->{$tab} = $ntab; - } + + my $file_trans = $self->{po_path_lang} . "/$file"; print "Write $file\n" if $self->{verbose}; - open my $fh, ">", $file_trans; - print $fh Dump($pref); + DumpFile($file_trans, $translated_pref); } } @@ -429,180 +303,6 @@ sub install_tmpl { } } - -sub update_tmpl { - my ($self, $files) = @_; - - say "Update templates" if $self->{verbose}; - for my $trans ( @{$self->{interface}} ) { - my @files = @$files; - my @nomarc = (); - print - " Update templates '$trans->{name}'\n", - " From: $trans->{dir}/en/\n", - " To : $self->{path_po}/$self->{lang}$trans->{suffix}\n" - if $self->{verbose}; - - my $trans_dir = join("/en/ -i ",split(" ",$trans->{dir}))."/en/"; # multiple source dirs - # if processing MARC po file, only use corresponding files - my $marc = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":""; # for MARC translations - # if not processing MARC po file, ignore all MARC files - @nomarc = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ ); # hardcoded MARC variants - - system - "$self->{process} update " . - "-i $trans_dir " . - "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " . - "$marc " . - ( @files ? ' -f ' . join ' -f ', @files : '') . - ( @nomarc ? ' -n ' . join ' -n ', @nomarc : ''); - } -} - - -sub create_prefs { - my $self = shift; - - if ( -e $self->po_filename("-pref.po") ) { - say "Preferences .po file already exists. Delete it if you want to recreate it."; - return; - } - $self->get_po_from_prefs(); - $self->save_po(); -} - -sub get_po_from_target { - my $self = shift; - my $target = shift; - - my $po; - my $po_head = new Locale::PO; - $po_head->{msgid} = "\"\""; - $po_head->{msgstr} = "". - "Project-Id-Version: Koha Project - Installation files\\n" . - "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\n" . - "Last-Translator: FULL NAME \\n" . - "Language-Team: Koha Translation Team\\n" . - "Language: ".$self->{lang}."\\n" . - "MIME-Version: 1.0\\n" . - "Content-Type: text/plain; charset=UTF-8\\n" . - "Content-Transfer-Encoding: 8bit\\n"; - - my @dirs = @{ $target->{dirs} }; - my $intradir = $self->{context}->config('intranetdir'); - for my $dir ( @dirs ) { # each dir - opendir( my $dh, "$intradir/$dir" ) or die ("Can't open $intradir/$dir"); - my @filelist = grep { $_ =~ m/\.yml/ } readdir($dh); # Just yaml files - close($dh); - for my $file ( @filelist ) { # each file - my $yaml = LoadFile( "$intradir/$dir/$file" ); - my @tables = @{ $yaml->{'tables'} }; - my $tablec; - for my $table ( @tables ) { # each table - $tablec++; - my $table_name = ( keys %$table )[0]; - my @translatable = @{ $table->{$table_name}->{translatable} }; - my @rows = @{ $table->{$table_name}->{rows} }; - my @multiline = @{ $table->{$table_name}->{'multiline'} }; # to check multiline values - my $rowc; - for my $row ( @rows ) { # each row - $rowc++; - for my $field ( @translatable ) { # each field - if ( @multiline and grep { $_ eq $field } @multiline ) { # multiline fields, only notices ATM - my $mulc; - foreach my $line ( @{$row->{$field}} ) { - $mulc++; - next if ( $line =~ /^(\s*<.*?>\s*$|^\s*\[.*?\]\s*|\s*)$/ ); # discard pure html, TT, empty - $line =~ s/(<<.*?>>|\[\%.*?\%\]|<.*?>)/\%s/g; # put placeholders - next if ( $line =~ /^(\s|%s|-|[[:punct:]]|\(|\))*$/ or length($line) < 2 ); # discard non strings - if ( not $po->{ $line } ) { - my $msg = new Locale::PO( - -msgid => $line, -msgstr => '', - -reference => "$dir/$file:$table_name:$tablec:row:$rowc:mul:$mulc" ); - $po->{ $line } = $msg; - } - } - } else { - if ( defined $row->{$field} and length($row->{$field}) > 1 # discard null values and small strings - and not $po->{ $row->{$field} } ) { - my $msg = new Locale::PO( - -msgid => $row->{$field}, -msgstr => '', - -reference => "$dir/$file:$table_name:$tablec:row:$rowc" ); - $po->{ $row->{$field} } = $msg; - } - } - } - } - } - my $desccount; - for my $description ( @{ $yaml->{'description'} } ) { - $desccount++; - if ( length($description) > 1 and not $po->{ $description } ) { - my $msg = new Locale::PO( - -msgid => $description, -msgstr => '', - -reference => "$dir/$file:description:$desccount" ); - $po->{ $description } = $msg; - } - } - } - } - $po->{''} = $po_head if ( $po ); - - return $po; -} - -sub create_installer { - my $self = shift; - return unless ( $self->{installer} ); - - say "Create installer translation files\n" if $self->{verbose}; - - my @targets = @{ $self->{installer} }; # each installer target (common,marc21,unimarc) - - for my $target ( @targets ) { - if ( -e $self->po_filename( $target->{suffix} ) ) { - say "$self->{lang}$target->{suffix} file already exists. Delete it if you want to recreate it."; - return; - } - } - - for my $target ( @targets ) { - my $po = get_po_from_target( $self, $target ); - # create output file only if there is something to write - if ( $po ) { - my $po_file = $self->po_filename( $target->{suffix} ); - Locale::PO->save_file_fromhash( $po_file, $po ); - say "Saved in file: ", $po_file if $self->{verbose}; - } - } -} - -sub update_installer { - my $self = shift; - return unless ( $self->{installer} ); - - say "Update installer translation files\n" if $self->{verbose}; - - my @targets = @{ $self->{installer} }; # each installer target (common,marc21,unimarc) - - for my $target ( @targets ) { - return unless ( -e $self->po_filename( $target->{suffix} ) ); - my $po = get_po_from_target( $self, $target ); - # update file only if there is something to update - if ( $po ) { - my ( $fh, $po_temp ) = tempfile(); - binmode( $fh, ":encoding(UTF-8)" ); - Locale::PO->save_file_fromhash( $po_temp, $po ); - my $po_file = $self->po_filename( $target->{suffix} ); - eval { - my $st = system($self->{msgmerge}." ".($self->{verbose}?'':'-q'). - " -s $po_file $po_temp -o - | ".$self->{msgattrib}." --no-obsolete -o $po_file"); - }; - say "Updated file: ", $po_file if $self->{verbose}; - } - } -} - sub translate_yaml { my $self = shift; my $target = shift; @@ -716,35 +416,6 @@ sub install_installer { } } -sub create_tmpl { - my ($self, $files) = @_; - - say "Create templates\n" if $self->{verbose}; - for my $trans ( @{$self->{interface}} ) { - my @files = @$files; - my @nomarc = (); - print - " Create templates .po files for '$trans->{name}'\n", - " From: $trans->{dir}/en/\n", - " To : $self->{path_po}/$self->{lang}$trans->{suffix}\n" - if $self->{verbose}; - - my $trans_dir = join("/en/ -i ",split(" ",$trans->{dir}))."/en/"; # multiple source dirs - # if processing MARC po file, only use corresponding files - my $marc = ( $trans->{name} =~ /MARC/ )?"-m \"$trans->{name}\"":""; # for MARC translations - # if not processing MARC po file, ignore all MARC files - @nomarc = ( 'marc21', 'unimarc', 'normarc' ) if ( $trans->{name} !~ /MARC/ ); # hardcoded MARC variants - - system - "$self->{process} create " . - "-i $trans_dir " . - "-s $self->{path_po}/$self->{lang}$trans->{suffix} -r " . - "$marc " . - ( @files ? ' -f ' . join ' -f ', @files : '') . - ( @nomarc ? ' -n ' . join ' -n ', @nomarc : ''); - } -} - sub locale_name { my $self = shift; @@ -758,250 +429,6 @@ sub locale_name { return $locale; } -sub create_messages { - my $self = shift; - - my $pot = "$Bin/$self->{domain}.pot"; - my $po = "$self->{path_po}/$self->{lang}-messages.po"; - my $js_pot = "$self->{domain}-js.pot"; - my $js_po = "$self->{path_po}/$self->{lang}-messages-js.po"; - - unless ( -f $pot && -f $js_pot ) { - $self->extract_messages(); - } - - say "Create messages ($self->{lang})" if $self->{verbose}; - my $locale = $self->locale_name(); - system "$self->{msginit} -i $pot -o $po -l $locale --no-translator 2> /dev/null"; - warn "Problems creating $pot ".$? if ( $? == -1 ); - system "$self->{msginit} -i $js_pot -o $js_po -l $locale --no-translator 2> /dev/null"; - warn "Problems creating $js_pot ".$? if ( $? == -1 ); - - # If msginit failed to correctly set Plural-Forms, set a default one - system "$self->{sed} --in-place " - . "--expression='s/Plural-Forms: nplurals=INTEGER; plural=EXPRESSION/Plural-Forms: nplurals=2; plural=(n != 1)/' " - . "$po $js_po"; -} - -sub update_messages { - my $self = shift; - - my $pot = "$Bin/$self->{domain}.pot"; - my $po = "$self->{path_po}/$self->{lang}-messages.po"; - my $js_pot = "$self->{domain}-js.pot"; - my $js_po = "$self->{path_po}/$self->{lang}-messages-js.po"; - - unless ( -f $pot && -f $js_pot ) { - $self->extract_messages(); - } - - if ( -f $po && -f $js_pot ) { - say "Update messages ($self->{lang})" if $self->{verbose}; - system "$self->{msgmerge} --backup=off --quiet -U $po $pot"; - system "$self->{msgmerge} --backup=off --quiet -U $js_po $js_pot"; - } else { - $self->create_messages(); - } -} - -sub extract_messages_from_templates { - my ($self, $tempdir, $type, @files) = @_; - - my $htdocs = $type eq 'intranet' ? 'intrahtdocs' : 'opachtdocs'; - my $dir = $self->{context}->config($htdocs); - my @keywords = qw(t tx tn txn tnx tp tpx tnp tnpx); - my $parser = Template::Parser->new(); - - foreach my $file (@files) { - say "Extract messages from $file" if $self->{verbose}; - my $template = read_file(File::Spec->catfile($dir, $file)); - - # No need to process a file that doesn't use the i18n.inc file. - next unless $template =~ /i18n\.inc/; - - my $data = $parser->parse($template); - unless ($data) { - warn "Error at $file : " . $parser->error(); - next; - } - - my $destfile = $type eq 'intranet' ? - File::Spec->catfile($tempdir, 'koha-tmpl', 'intranet-tmpl', $file) : - File::Spec->catfile($tempdir, 'koha-tmpl', 'opac-tmpl', $file); - - make_path(dirname($destfile)); - open my $fh, '>', $destfile; - - my @blocks = ($data->{BLOCK}, values %{ $data->{DEFBLOCKS} }); - foreach my $block (@blocks) { - my $document = PPI::Document->new(\$block); - - # [% t('foo') %] is compiled to - # $output .= $stash->get(['t', ['foo']]); - # We try to find all nodes corresponding to keyword (here 't') - my $nodes = $document->find(sub { - my ($topnode, $element) = @_; - - # Filter out non-valid keywords - return 0 unless ($element->isa('PPI::Token::Quote::Single')); - return 0 unless (grep {$element->content eq qq{'$_'}} @keywords); - - # keyword (e.g. 't') should be the first element of the arrayref - # passed to $stash->get() - return 0 if $element->sprevious_sibling; - - return 0 unless $element->snext_sibling - && $element->snext_sibling->snext_sibling - && $element->snext_sibling->snext_sibling->isa('PPI::Structure::Constructor'); - - # Check that it's indeed a call to $stash->get() - my $statement = $element->statement->parent->statement->parent->statement; - return 0 unless grep { $_->isa('PPI::Token::Symbol') && $_->content eq '$stash' } $statement->children; - return 0 unless grep { $_->isa('PPI::Token::Operator') && $_->content eq '->' } $statement->children; - return 0 unless grep { $_->isa('PPI::Token::Word') && $_->content eq 'get' } $statement->children; - - return 1; - }); - - next unless $nodes; - - # Write the Perl equivalent of calls to t* functions family, so - # xgettext can extract the strings correctly - foreach my $node (@$nodes) { - my @args = map { - $_->significant && !$_->isa('PPI::Token::Operator') ? $_->content : () - } $node->snext_sibling->snext_sibling->find_first('PPI::Statement')->children; - - my $keyword = $node->content; - $keyword =~ s/^'t(.*)'$/__$1/; - - # Only keep required args to have a clean output - my @required_args = shift @args; - push @required_args, shift @args if $keyword =~ /n/; - push @required_args, shift @args if $keyword =~ /p/; - - say $fh "$keyword(" . join(', ', @required_args) . ");"; - } - - } - - close $fh; - } - - return $tempdir; -} - -sub extract_messages { - my $self = shift; - - say "Extract messages into POT file" if $self->{verbose}; - - my $intranetdir = $self->{context}->config('intranetdir'); - my $opacdir = $self->{context}->config('opacdir'); - - # Find common ancestor directory - my @intranetdirs = File::Spec->splitdir($intranetdir); - my @opacdirs = File::Spec->splitdir($opacdir); - my @basedirs; - while (@intranetdirs and @opacdirs) { - my ($dir1, $dir2) = (shift @intranetdirs, shift @opacdirs); - last if $dir1 ne $dir2; - push @basedirs, $dir1; - } - my $basedir = File::Spec->catdir(@basedirs); - - my @files_to_scan; - my @directories_to_scan = ('.'); - my @blacklist = map { File::Spec->catdir(@intranetdirs, $_) } qw(blib koha-tmpl skel tmp t); - while (@directories_to_scan) { - my $dir = shift @directories_to_scan; - opendir DIR, File::Spec->catdir($basedir, $dir) or die "Unable to open $dir: $!"; - foreach my $entry (readdir DIR) { - next if $entry =~ /^\./; - my $relentry = File::Spec->catfile($dir, $entry); - my $abspath = File::Spec->catfile($basedir, $relentry); - if (-d $abspath and not grep { $_ eq $relentry } @blacklist) { - push @directories_to_scan, $relentry; - } elsif (-f $abspath and $relentry =~ /\.(pl|pm)$/) { - push @files_to_scan, $relentry; - } - } - } - - my $intrahtdocs = $self->{context}->config('intrahtdocs'); - my $opachtdocs = $self->{context}->config('opachtdocs'); - - my @intranet_tt_files; - find(sub { - if ($File::Find::dir =~ m|/en/| && $_ =~ m/\.(tt|inc)$/) { - my $filename = $File::Find::name; - $filename =~ s|^$intrahtdocs/||; - push @intranet_tt_files, $filename; - } - }, $intrahtdocs); - - my @opac_tt_files; - find(sub { - if ($File::Find::dir =~ m|/en/| && $_ =~ m/\.(tt|inc)$/) { - my $filename = $File::Find::name; - $filename =~ s|^$opachtdocs/||; - push @opac_tt_files, $filename; - } - }, $opachtdocs); - - my $tempdir = tempdir('Koha-translate-XXXX', TMPDIR => 1, CLEANUP => 1); - $self->extract_messages_from_templates($tempdir, 'intranet', @intranet_tt_files); - $self->extract_messages_from_templates($tempdir, 'opac', @opac_tt_files); - - @intranet_tt_files = map { File::Spec->catfile('koha-tmpl', 'intranet-tmpl', $_) } @intranet_tt_files; - @opac_tt_files = map { File::Spec->catfile('koha-tmpl', 'opac-tmpl', $_) } @opac_tt_files; - my @tt_files = grep { -e File::Spec->catfile($tempdir, $_) } @intranet_tt_files, @opac_tt_files; - - push @files_to_scan, @tt_files; - - my $xgettext_common_args = "--force-po --from-code=UTF-8 " - . "--package-name=Koha --package-version='' " - . "-k -k__ -k__x -k__n:1,2 -k__nx:1,2 -k__xn:1,2 -k__p:1c,2 " - . "-k__px:1c,2 -k__np:1c,2,3 -k__npx:1c,2,3 -kN__ -kN__n:1,2 " - . "-kN__p:1c,2 -kN__np:1c,2,3 "; - my $xgettext_cmd = "$self->{xgettext} -L Perl $xgettext_common_args " - . "-o $Bin/$self->{domain}.pot -D $tempdir -D $basedir"; - $xgettext_cmd .= " $_" foreach (@files_to_scan); - - if (system($xgettext_cmd) != 0) { - die "system call failed: $xgettext_cmd"; - } - - my @js_dirs = ( - "$intrahtdocs/prog/js", - "$opachtdocs/bootstrap/js", - ); - - my @js_files; - find(sub { - if ($_ =~ m/\.js$/) { - my $filename = $File::Find::name; - $filename =~ s|^$intranetdir/||; - push @js_files, $filename; - } - }, @js_dirs); - - $xgettext_cmd = "$self->{xgettext} -L JavaScript $xgettext_common_args " - . "-o $Bin/$self->{domain}-js.pot -D $intranetdir"; - $xgettext_cmd .= " $_" foreach (@js_files); - - if (system($xgettext_cmd) != 0) { - die "system call failed: $xgettext_cmd"; - } - - my $replace_charset_cmd = "$self->{sed} --in-place " . - "--expression='s/charset=CHARSET/charset=UTF-8/' " . - "$Bin/$self->{domain}.pot $Bin/$self->{domain}-js.pot"; - if (system($replace_charset_cmd) != 0) { - die "system call failed: $replace_charset_cmd"; - } -} - sub install_messages { my ($self) = @_; @@ -1012,8 +439,9 @@ sub install_messages { my $js_pofile = "$self->{path_po}/$self->{lang}-messages-js.po"; unless ( -f $pofile && -f $js_pofile ) { - $self->create_messages(); + die "PO files for language '$self->{lang}' do not exist"; } + say "Install messages ($locale)" if $self->{verbose}; make_path($modir); system "$self->{msgfmt} -o $mofile $pofile"; @@ -1035,13 +463,6 @@ sub install_messages { } } -sub remove_pot { - my $self = shift; - - unlink "$Bin/$self->{domain}.pot"; - unlink "$Bin/$self->{domain}-js.pot"; -} - sub compress { my ($self, $files) = @_; my @langs = $self->{lang} ? ($self->{lang}) : $self->get_all_langs(); @@ -1074,11 +495,15 @@ sub install { my ($self, $files) = @_; return unless $self->{lang}; $self->uncompress(); - $self->install_tmpl($files) unless $self->{pref_only}; - $self->install_prefs(); - $self->install_messages(); - $self->remove_pot(); - $self->install_installer(); + + if ($self->{pref_only}) { + $self->install_prefs(); + } else { + $self->install_tmpl($files); + $self->install_prefs(); + $self->install_messages(); + $self->install_installer(); + } } @@ -1090,34 +515,6 @@ sub get_all_langs { @files = map { $_ =~ s/-pref.(po|po.gz)$//r } @files; } - -sub update { - my ($self, $files) = @_; - my @langs = $self->{lang} ? ($self->{lang}) : $self->get_all_langs(); - for my $lang ( @langs ) { - $self->set_lang( $lang ); - $self->uncompress(); - $self->update_tmpl($files) unless $self->{pref_only}; - $self->update_prefs(); - $self->update_messages(); - $self->update_installer(); - } - $self->remove_pot(); -} - - -sub create { - my ($self, $files) = @_; - return unless $self->{lang}; - $self->create_tmpl($files) unless $self->{pref_only}; - $self->create_prefs(); - $self->create_messages(); - $self->remove_pot(); - $self->create_installer(); -} - - - 1; diff --git a/misc/translator/po/dz-pref.po b/misc/translator/po/dz-pref.po index 8618b78857..8f543cc7f7 100644 --- a/misc/translator/po/dz-pref.po +++ b/misc/translator/po/dz-pref.po @@ -1,5 +1,13 @@ msgid "" -msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME \\nLanguage-Team: Koha Translate List \\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n" +"Last-Translator: FULL NAME \n" +"Language-Team: Koha Translate List \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Plural-Forms: nplurals=2; plural=(n > 1);\n" # Accounting msgid "accounting.pref" diff --git a/misc/translator/po/gd-pref.po b/misc/translator/po/gd-pref.po index 8618b78857..8f543cc7f7 100644 --- a/misc/translator/po/gd-pref.po +++ b/misc/translator/po/gd-pref.po @@ -1,5 +1,13 @@ msgid "" -msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME \\nLanguage-Team: Koha Translate List \\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n" +"Last-Translator: FULL NAME \n" +"Language-Team: Koha Translate List \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Plural-Forms: nplurals=2; plural=(n > 1);\n" # Accounting msgid "accounting.pref" diff --git a/misc/translator/po/lv-pref.po b/misc/translator/po/lv-pref.po index 8618b78857..8f543cc7f7 100644 --- a/misc/translator/po/lv-pref.po +++ b/misc/translator/po/lv-pref.po @@ -1,5 +1,13 @@ msgid "" -msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME \\nLanguage-Team: Koha Translate List \\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n" +"Last-Translator: FULL NAME \n" +"Language-Team: Koha Translate List \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Plural-Forms: nplurals=2; plural=(n > 1);\n" # Accounting msgid "accounting.pref" diff --git a/misc/translator/po/te-pref.po b/misc/translator/po/te-pref.po index 8618b78857..8f543cc7f7 100644 --- a/misc/translator/po/te-pref.po +++ b/misc/translator/po/te-pref.po @@ -1,5 +1,13 @@ msgid "" -msgstr "Project-Id-Version: PACKAGE VERSION\\nPO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\\nLast-Translator: FULL NAME \\nLanguage-Team: Koha Translate List \\nMIME-Version: 1.0\\nContent-Type: text/plain; charset=UTF-8\\nContent-Transfer-Encoding: 8bit\\nPlural-Forms: nplurals=2; plural=(n > 1);\\n" +msgstr "" +"Project-Id-Version: PACKAGE VERSION\n" +"PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n" +"Last-Translator: FULL NAME \n" +"Language-Team: Koha Translate List \n" +"MIME-Version: 1.0\n" +"Content-Type: text/plain; charset=UTF-8\n" +"Content-Transfer-Encoding: 8bit\n" +"Plural-Forms: nplurals=2; plural=(n > 1);\n" # Accounting msgid "accounting.pref" diff --git a/misc/translator/tmpl_process3.pl b/misc/translator/tmpl_process3.pl index 9221430c35..3c04618915 100755 --- a/misc/translator/tmpl_process3.pl +++ b/misc/translator/tmpl_process3.pl @@ -204,11 +204,9 @@ sub usage { my($exitcode) = @_; my $h = $exitcode? *STDERR: *STDOUT; print $h </POTFILES.in - for my $input (@in_files) { - print $tmph1 "$input\n"; - } - close $tmph1; - warn "I $charset_in O $charset_out" unless ( $quiet ); - # Generate the specified po file ($str_file) - $st = system ($xgettext, '-s', '-f', $tmpfile1, '-o', $tmpfile2, - (defined $charset_in? ('-I', $charset_in): ()), - (defined $charset_out? ('-O', $charset_out): ()) - ); - # Run msgmerge so that the pot file looks like a real pot file - # We need to help msgmerge a bit by pre-creating a dummy po file that has - # the headers and the "" msgid & msgstr. It will fill in the rest. - if ($st == 0) { - # Merge the temporary "pot file" with the specified po file ($str_file) - # FIXME: msgmerge(1) is a Unix dependency - # FIXME: need to check the return value - unless (-f $str_file) { - open(my $infh, '<', $tmpfile2); - open(my $outfh, '>', $str_file); - while (<$infh>) { - print $outfh $_; - last if /^\n/s; - } - close $infh; - close $outfh; - } - $st = system("msgmerge ".($quiet?'-q':'')." -s $str_file $tmpfile2 -o - | msgattrib --no-obsolete -o $str_file"); - } else { - error_normal("Text extraction failed: $xgettext: $!\n", undef); - error_additional("Will not run msgmerge\n", undef); - } - unlink $tmpfile1 || warn_normal("$tmpfile1: unlink failed: $!\n", undef); - unlink $tmpfile2 || warn_normal("$tmpfile2: unlink failed: $!\n", undef); - -} elsif ($action eq 'update') { - my($tmph1, $tmpfile1) = tmpnam(); - my($tmph2, $tmpfile2) = tmpnam(); - close $tmph2; # We just want a name - # Generate the temporary file that acts as /POTFILES.in - for my $input (@in_files) { - print $tmph1 "$input\n"; - } - close $tmph1; - # Generate the temporary file that acts as /.pot - $st = system($xgettext, '-s', '-f', $tmpfile1, '-o', $tmpfile2, - '--po-mode', - (defined $charset_in? ('-I', $charset_in): ()), - (defined $charset_out? ('-O', $charset_out): ())); - if ($st == 0) { - # Merge the temporary "pot file" with the specified po file ($str_file) - # FIXME: msgmerge(1) is a Unix dependency - # FIXME: need to check the return value - if ( @filenames ) { - my ($tmph3, $tmpfile3) = tmpnam(); - $st = system("msgcat $str_file $tmpfile2 > $tmpfile3"); - $st = system("msgmerge ".($quiet?'-q':'')." -s $str_file $tmpfile3 -o - | msgattrib --no-obsolete -o $str_file") - unless $st; - } else { - $st = system("msgmerge ".($quiet?'-q':'')." -s $str_file $tmpfile2 -o - | msgattrib --no-obsolete -o $str_file"); - } - } else { - error_normal("Text extraction failed: $xgettext: $!\n", undef); - error_additional("Will not run msgmerge\n", undef); - } - unlink $tmpfile1 || warn_normal("$tmpfile1: unlink failed: $!\n", undef); - unlink $tmpfile2 || warn_normal("$tmpfile2: unlink failed: $!\n", undef); - -} elsif ($action eq 'install') { +if ($action eq 'install') { if(!defined($out_dir)) { usage_error("You must specify an output directory when using the install method."); } @@ -554,14 +465,6 @@ translation, it can be suppressed with the %0.0s notation. Using the PO format also means translators can add their own comments in the translation files, if necessary. -=item - - -Create, update, and install actions are all based on the -same scanner module. This ensures that update and install -have the same idea of what is a translatable string; -attribute names in tags, for example, will not be -accidentally translated. - =back =head1 NOTES @@ -569,22 +472,8 @@ accidentally translated. Anchors are represented by an > notation. The meaning of this non-standard notation might not be obvious. -The create action calls xgettext.pl to do the actual work; -the update action calls xgettext.pl, msgmerge(1) and msgattrib(1) -to do the actual work. - =head1 BUGS -xgettext.pl must be present in the current directory; both -msgmerge(1) and msgattrib(1) must also be present in the search path. -The script currently does not check carefully whether these -dependent commands are present. - -Locale::PO(3) has a lot of bugs. It can neither parse nor -generate GNU PO files properly; a couple of workarounds have -been written in TmplTokenizer and more is likely to be needed -(e.g., to get rid of the "Strange line" warning for #~). - This script may not work in Windows. There are probably some other bugs too, since this has not been @@ -592,12 +481,7 @@ tested very much. =head1 SEE ALSO -xgettext.pl, TmplTokenizer.pm, -msgmerge(1), Locale::PO(3), -translator_doc.txt - -http://www.saas.nsw.edu.au/koha_wiki/index.php?page=DifficultTerms =cut diff --git a/misc/translator/translate b/misc/translator/translate index f53dd464ff..08764f0b42 100755 --- a/misc/translator/translate +++ b/misc/translator/translate @@ -54,14 +54,13 @@ usage() if $#ARGV != 1 && $#ARGV != 0; my ($cmd, $lang) = @ARGV; $cmd = lc $cmd; -if ( $cmd =~ /^(create|install|update|compress|uncompress)$/ ) { +if ( $cmd =~ /^(install|compress|uncompress)$/ ) { my $installer = LangInstaller->new( $lang, $pref, $verbose ); - if ( $cmd ne 'create' and $lang and not grep( {$_ eq $lang} @{ $installer->{langs} } ) ) { + if ( $lang and not grep( {$_ eq $lang} @{ $installer->{langs} } ) ) { print "Unsupported language: $lang\n"; exit; } if ( $all ) { - usage() if $cmd eq 'create'; for my $lang ( @{$installer->{langs}} ) { $installer->set_lang( $lang ); $installer->$cmd(\@files); @@ -85,12 +84,9 @@ translate - Handle templates and preferences translation =head1 SYNOPSYS - translate create fr-FR - translate update fr-FR translate install fr-FR translate install fr-FR -f search -f memberentry translate -p install fr-FR - translate install translate compress [fr-FR] translate uncompress [fr-FR] @@ -98,7 +94,7 @@ translate - Handle templates and preferences translation In Koha, three categories of information are translated based on standard GNU .po files: opac templates pages, intranet templates and system preferences. The -script is a wrapper. It allows to quickly create/update/install .po files for a +script is a wrapper. It allows to quickly install .po files for a given language or for all available languages. =head1 USAGE @@ -107,38 +103,6 @@ Use the -v or --verbose parameter to make translator more verbose. =over -=item translate create F - -Create 3 .po files in F subdirectory: (1) from opac pages -templates, (2) intranet templates, and (3) from preferences. English 'en' -version of templates and preferences are used as references. - -=over - -=item F-opac-{theme}.po - -Contains extracted text from english (en) OPAC templates found in -/koha-tmpl/opac-tmpl/{theme}/en/ directory. - -=item F-intranet.po - -Contains extracted text from english (en) intranet templates found in -/koha-tmpl/intranet-tmpl/prog/en/ directory. - -=item F-pref.po - -Contains extracted text from english (en) preferences. They are found in files -located in /koha-tmpl/intranet-tmpl/prog/en/admin/preferences -directory. - -=back - -=item translate [-p] update F - -Update .po files in F directory, named F-*.po. Without F, all -available languages are updated. With -p option, only preferences .po file is -updated. - =item translate [-p|-f] install F Use .po files to translate the english version of templates and preferences files diff --git a/misc/translator/xgettext-installer b/misc/translator/xgettext-installer new file mode 100755 index 0000000000..6e415898a6 --- /dev/null +++ b/misc/translator/xgettext-installer @@ -0,0 +1,143 @@ +#!/usr/bin/perl + +=head1 NAME + +xgettext-installer - extract translatable strings from installer YAML files + +=head1 SYNOPSIS + +xgettext-installer [OPTION] [INPUTFILE]... + +=head1 OPTIONS + +=over + +=item B<-f, --files-from=FILE> + +get list of input files from FILE + +=item B<-o, --output=FILE> + +write output to the specified file + +=item B<-h, --help> + +display this help and exit + +=back + +=cut + +use Modern::Perl; + +use Getopt::Long; +use Locale::PO; +use Pod::Usage; +use YAML::Syck qw(LoadFile); + +$YAML::Syck::ImplicitTyping = 1; + +my $output = 'messages.pot'; +my $files_from; +my $help; + +GetOptions( + 'output=s' => \$output, + 'files-from=s' => \$files_from, + 'help' => \$help, +) or pod2usage(-verbose => 1, -exitval => 2); + +if ($help) { + pod2usage(-verbose => 1, -exitval => 0); +} + +my @files = @ARGV; +if ($files_from) { + open(my $fh, '<', $files_from) or die "Cannot open $files_from: $!"; + push @files, <$fh>; + chomp @files; + close $fh; +} + +my $pot = { + '' => Locale::PO->new( + -msgid => '', + -msgstr => + "Project-Id-Version: Koha\n" + . "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n" + . "Last-Translator: FULL NAME \n" + . "Language-Team: Koha Translate List \n" + . "MIME-Version: 1.0\n" + . "Content-Type: text/plain; charset=UTF-8\n" + . "Content-Transfer-Encoding: 8bit\n" + ), +}; + +for my $file (@files) { + my $yaml = LoadFile($file); + my @tables = @{ $yaml->{'tables'} }; + + my $tablec = 0; + for my $table (@tables) { + $tablec++; + + my $table_name = ( keys %$table )[0]; + my @translatable = @{ $table->{$table_name}->{translatable} }; + my @rows = @{ $table->{$table_name}->{rows} }; + my @multiline = @{ $table->{$table_name}->{'multiline'} }; + + my $rowc = 0; + for my $row (@rows) { + $rowc++; + + for my $field (@translatable) { + if ( @multiline and grep { $_ eq $field } @multiline ) { + # multiline fields, only notices ATM + my $mulc; + foreach my $line ( @{ $row->{$field} } ) { + $mulc++; + + # discard pure html, TT, empty + next if ( $line =~ /^(\s*<.*?>\s*$|^\s*\[.*?\]\s*|\s*)$/ ); + + # put placeholders + $line =~ s/(<<.*?>>|\[\%.*?\%\]|<.*?>)/\%s/g; + + # discard non strings + next if ( $line =~ /^(\s|%s|-|[[:punct:]]|\(|\))*$/ or length($line) < 2 ); + if ( not $pot->{$line} ) { + my $msg = new Locale::PO( + -msgid => $line, + -msgstr => '', + -reference => "$file:$table_name:$tablec:row:$rowc:mul:$mulc" + ); + $pot->{$line} = $msg; + } + } + } elsif (defined $row->{$field} && length($row->{$field}) > 1 && !$pot->{ $row->{$field} }) { + my $msg = new Locale::PO( + -msgid => $row->{$field}, + -msgstr => '', + -reference => "$file:$table_name:$tablec:row:$rowc" + ); + $pot->{ $row->{$field} } = $msg; + } + } + } + } + + my $desccount = 0; + for my $description ( @{ $yaml->{'description'} } ) { + $desccount++; + if ( length($description) > 1 and not $pot->{$description} ) { + my $msg = new Locale::PO( + -msgid => $description, + -msgstr => '', + -reference => "$file:description:$desccount" + ); + $pot->{$description} = $msg; + } + } +} + +Locale::PO->save_file_fromhash($output, $pot); diff --git a/misc/translator/xgettext-pref b/misc/translator/xgettext-pref new file mode 100755 index 0000000000..baf0381cc4 --- /dev/null +++ b/misc/translator/xgettext-pref @@ -0,0 +1,134 @@ +#!/usr/bin/perl + +=head1 NAME + +xgettext-pref - extract translatable strings from system preferences YAML files + +=head1 SYNOPSIS + +xgettext-pref [OPTION] [INPUTFILE]... + +=head1 OPTIONS + +=over + +=item B<-f, --files-from=FILE> + +get list of input files from FILE + +=item B<-o, --output=FILE> + +write output to the specified file + +=item B<-h, --help> + +display this help and exit + +=back + +=cut + +use Modern::Perl; + +use Getopt::Long; +use Locale::PO; +use Pod::Usage; +use YAML::Syck qw(LoadFile); + +$YAML::Syck::ImplicitTyping = 1; + +my $output = 'messages.pot'; +my $files_from; +my $help; + +GetOptions( + 'output=s' => \$output, + 'files-from=s' => \$files_from, + 'help' => \$help, +) or pod2usage(-verbose => 1, -exitval => 2); + +if ($help) { + pod2usage(-verbose => 1, -exitval => 0); +} + +my @files = @ARGV; +if ($files_from) { + open(my $fh, '<', $files_from) or die "Cannot open $files_from: $!"; + push @files, <$fh>; + chomp @files; + close $fh; +} + +my $pot = { + '' => Locale::PO->new( + -msgid => '', + -msgstr => "Project-Id-Version: Koha\n" + . "PO-Revision-Date: YEAR-MO-DA HO:MI +ZONE\n" + . "Last-Translator: FULL NAME \n" + . "Language-Team: Koha Translate List \n" + . "MIME-Version: 1.0\n" + . "Content-Type: text/plain; charset=UTF-8\n" + . "Content-Transfer-Encoding: 8bit\n" + ), +}; + +for my $file (@files) { + my $pref = LoadFile($file); + while ( my ($tab, $tab_content) = each %$pref ) { + add_po($file, undef, $tab); + + if ( ref($tab_content) eq 'ARRAY' ) { + add_prefs( $file, $tab, $tab_content ); + } else { + while ( my ($section, $sysprefs) = each %$tab_content ) { + my $context = "$tab > $section"; + add_po($file, $tab, $section); + add_prefs( $file, $context, $sysprefs ); + } + } + } +} + +Locale::PO->save_file_fromhash($output, $pot); + +sub add_prefs { + my ($file, $context, $prefs) = @_; + + for my $pref (@$prefs) { + my $pref_name = ''; + for my $element (@$pref) { + if ( ref($element) eq 'HASH' ) { + $pref_name = $element->{pref}; + last; + } + } + for my $element (@$pref) { + if ( ref($element) eq 'HASH' ) { + while ( my ( $key, $value ) = each(%$element) ) { + next unless $key eq 'choices' or $key eq 'multiple'; + next unless ref($value) eq 'HASH'; + for my $ckey ( keys %$value ) { + add_po( $file, "$context > $pref_name", $value->{$ckey} ); + } + } + } + elsif ($element) { + add_po( $file, "$context > $pref_name", $element ); + } + } + } +} + +sub add_po { + my ( $reference, $msgctxt, $msgid ) = @_; + + return unless $msgid; + + my $key = ($msgctxt // '') . ";$msgid"; + $pot->{$key} = Locale::PO->new( + -reference => $reference, + -msgctxt => $msgctxt, + -msgid => $msgid, + -msgstr => '', + ); +} diff --git a/misc/translator/xgettext-tt2 b/misc/translator/xgettext-tt2 new file mode 100755 index 0000000000..d083a458db --- /dev/null +++ b/misc/translator/xgettext-tt2 @@ -0,0 +1,41 @@ +#!/usr/bin/perl + +use Modern::Perl; + +my $xgettext = Locale::XGettext::TT2::Koha->newFromArgv(\@ARGV); +$xgettext->setOption('plug_in', ''); +$xgettext->run; +$xgettext->output; + +package Locale::XGettext::TT2::Koha; + +use parent 'Locale::XGettext::TT2'; + +sub defaultKeywords { + return [ + 't:1', + 'tx:1', + 'tn:1,2', + 'tnx:1,2', + 'txn:1,2', + 'tp:1c,2', + 'tpx:1c,2', + 'tnp:1c,2,3', + 'tnpx:1c,2,3', + ]; +} + +sub defaultFlags { + return [ + 'tx:1:perl-brace-format', + 'tnx:1:perl-brace-format', + 'tnx:2:perl-brace-format', + 'txn:1:perl-brace-format', + 'txn:2:perl-brace-format', + 'tpx:2:perl-brace-format', + 'tnpx:2:perl-brace-format', + 'tnpx:3:perl-brace-format', + ], +} + +1; diff --git a/misc/translator/xgettext.pl b/misc/translator/xgettext.pl index f9ba3bf8d0..180bb543d6 100755 --- a/misc/translator/xgettext.pl +++ b/misc/translator/xgettext.pl @@ -173,7 +173,7 @@ EOF print $OUTPUT <\\n" diff --git a/package.json b/package.json index 5483e873c1..0837784070 100644 --- a/package.json +++ b/package.json @@ -9,9 +9,12 @@ "dependencies": { "gulp": "^4.0.2", "gulp-autoprefixer": "^4.0.0", + "gulp-concat-po": "^1.0.0", "gulp-cssnano": "^2.1.2", + "gulp-exec": "^4.0.0", "gulp-sass": "^3.1.0", "gulp-sourcemaps": "^2.6.1", + "merge-stream": "^2.0.0", "minimist": "^1.2.5" }, "devDependencies": {}, diff --git a/t/LangInstaller.t b/t/LangInstaller.t deleted file mode 100644 index 6e122b368e..0000000000 --- a/t/LangInstaller.t +++ /dev/null @@ -1,109 +0,0 @@ -use Modern::Perl; - -use FindBin '$Bin'; -use lib "$Bin/../misc/translator"; - -use Test::More tests => 39; -use File::Temp qw(tempdir); -use File::Slurp; -use Locale::PO; - -use t::lib::Mocks; - -use_ok('LangInstaller'); - -my $installer = LangInstaller->new(); - -my $tempdir = tempdir(CLEANUP => 0); -t::lib::Mocks::mock_config('intrahtdocs', "$Bin/LangInstaller/templates"); -my @files = ('simple.tt'); -$installer->extract_messages_from_templates($tempdir, 'intranet', @files); - -my $tempfile = "$tempdir/koha-tmpl/intranet-tmpl/simple.tt"; -ok(-e $tempfile, 'it has created a temporary file simple.tt'); -SKIP: { - skip "simple.tt does not exist", 37 unless -e $tempfile; - - my $output = read_file($tempfile); - my $expected_output = <<'EOF'; -__('hello'); -__x('hello {name}'); -__n('item', 'items'); -__nx('{count} item', '{count} items'); -__p('context', 'hello'); -__px('context', 'hello {name}'); -__np('context', 'item', 'items'); -__npx('context', '{count} item', '{count} items'); -__npx('context', '{count} item', '{count} items'); -__x('status is {status}'); -__('active'); -__('inactive'); -__('Inside block'); -EOF - - is($output, $expected_output, "Output of extract_messages_from_templates is as expected"); - - my $xgettext_cmd = "xgettext -L Perl --from-code=UTF-8 " - . "--package-name=Koha --package-version='' " - . "-k -k__ -k__x -k__n:1,2 -k__nx:1,2 -k__xn:1,2 -k__p:1c,2 " - . "-k__px:1c,2 -k__np:1c,2,3 -k__npx:1c,2,3 " - . "-o $tempdir/Koha.pot -D $tempdir koha-tmpl/intranet-tmpl/simple.tt"; - - system($xgettext_cmd); - my $pot = Locale::PO->load_file_asarray("$tempdir/Koha.pot"); - - my @expected = ( - { - msgid => '"hello"', - }, - { - msgid => '"hello {name}"', - }, - { - msgid => '"item"', - msgid_plural => '"items"', - }, - { - msgid => '"{count} item"', - msgid_plural => '"{count} items"', - }, - { - msgid => '"hello"', - msgctxt => '"context"', - }, - { - msgid => '"hello {name}"', - msgctxt => '"context"', - }, - { - msgid => '"item"', - msgid_plural => '"items"', - msgctxt => '"context"', - }, - { - msgid => '"{count} item"', - msgid_plural => '"{count} items"', - msgctxt => '"context"', - }, - { - msgid => '"status is {status}"', - }, - { - msgid => '"active"', - }, - { - msgid => '"inactive"', - }, - { - msgid => '"Inside block"', - }, - ); - - for (my $i = 0; $i < @expected; $i++) { - for my $key (qw(msgid msgid_plural msgctxt)) { - my $expected = $expected[$i]->{$key}; - my $expected_str = defined $expected ? $expected : 'not defined'; - is($pot->[$i + 1]->$key, $expected, "$i: $key is $expected_str"); - } - } -} diff --git a/t/misc/translator/sample.pref b/t/misc/translator/sample.pref new file mode 100644 index 0000000000..c42692b44c --- /dev/null +++ b/t/misc/translator/sample.pref @@ -0,0 +1,14 @@ +Section: + Subsection: + - + - pref: SamplePref + choices: + on: Do + off: Do not do + - that thing + - + - pref: MultiplePref + multiple: + foo: Foo ツ + bar: Bar + baz: Baz diff --git a/t/LangInstaller/templates/simple.tt b/t/misc/translator/sample.tt similarity index 96% rename from t/LangInstaller/templates/simple.tt rename to t/misc/translator/sample.tt index dbc4ce3882..3fd4fca2a5 100644 --- a/t/LangInstaller/templates/simple.tt +++ b/t/misc/translator/sample.tt @@ -1,6 +1,6 @@ [% USE raw %] [% PROCESS 'i18n.inc' %] -[% t('hello') | $raw %] +[% t('hello ツ') | $raw %] [% tx('hello {name}', { name = 'Bob' }) | $raw %] [% tn('item', 'items', count) | $raw %] [% tnx('{count} item', '{count} items', count, { count = count }) | $raw %] diff --git a/t/misc/translator/sample.yml b/t/misc/translator/sample.yml new file mode 100644 index 0000000000..d6d675f99a --- /dev/null +++ b/t/misc/translator/sample.yml @@ -0,0 +1,15 @@ +description: + - "Sample installer file" + +tables: + - table1: + translatable: [ column1, column2 ] + multiline: [ column2 ] + rows: + - column1: foo ツ + column2: + - bar + - baz + column3: qux + column4: + - quux diff --git a/t/misc/translator/xgettext-installer.t b/t/misc/translator/xgettext-installer.t new file mode 100644 index 0000000000..404361088a --- /dev/null +++ b/t/misc/translator/xgettext-installer.t @@ -0,0 +1,32 @@ +#!/usr/bin/perl + +use Modern::Perl; + +use File::Slurp; +use File::Temp qw(tempdir); +use FindBin qw($Bin); +use Locale::PO; +use Test::More tests => 4; + +my $tempdir = tempdir(CLEANUP => 1); + +write_file("$tempdir/files", "$Bin/sample.yml"); + +my $xgettext_cmd = "$Bin/../../../misc/translator/xgettext-installer " + . "-o $tempdir/Koha.pot -f $tempdir/files"; + +system($xgettext_cmd); +my $pot = Locale::PO->load_file_asarray("$tempdir/Koha.pot"); + +my @expected = ( + { msgid => '"Sample installer file"' }, + { msgid => '"bar"' }, + { msgid => '"baz"' }, + { msgid => '"foo ツ"' }, +); + +for (my $i = 0; $i < @expected; $i++) { + my $expected = $expected[$i]->{msgid}; + my $expected_str = defined $expected ? $expected : 'not defined'; + is($pot->[$i + 1]->msgid, $expected, "$i: msgid is $expected_str"); +} diff --git a/t/misc/translator/xgettext-pref.t b/t/misc/translator/xgettext-pref.t new file mode 100644 index 0000000000..4482d67bba --- /dev/null +++ b/t/misc/translator/xgettext-pref.t @@ -0,0 +1,61 @@ +#!/usr/bin/perl + +use Modern::Perl; + +use File::Slurp; +use File::Temp qw(tempdir); +use FindBin qw($Bin); +use Locale::PO; +use Test::More tests => 16; + +my $tempdir = tempdir(CLEANUP => 1); + +write_file("$tempdir/files", "$Bin/sample.pref"); + +my $xgettext_cmd = "$Bin/../../../misc/translator/xgettext-pref " + . "-o $tempdir/Koha.pot -f $tempdir/files"; + +system($xgettext_cmd); +my $pot = Locale::PO->load_file_asarray("$tempdir/Koha.pot"); + +my @expected = ( + { + msgid => '"Section"', + }, + { + msgctxt => '"Section > Subsection > MultiplePref"', + msgid => '"Bar"', + }, + { + msgctxt => '"Section > Subsection > MultiplePref"', + msgid => '"Baz"', + }, + { + msgctxt => '"Section > Subsection > MultiplePref"', + msgid => '"Foo ツ"', + }, + { + msgctxt => '"Section > Subsection > SamplePref"', + msgid => '"Do"', + }, + { + msgctxt => '"Section > Subsection > SamplePref"', + msgid => '"Do not do"', + }, + { + msgctxt => '"Section > Subsection > SamplePref"', + msgid => '"that thing"', + }, + { + msgctxt => '"Section"', + msgid => '"Subsection"', + }, +); + +for (my $i = 0; $i < @expected; $i++) { + for my $key (qw(msgid msgctxt)) { + my $expected = $expected[$i]->{$key}; + my $expected_str = defined $expected ? $expected : 'not defined'; + is($pot->[$i + 1]->$key, $expected, "$i: $key is $expected_str"); + } +} diff --git a/t/misc/translator/xgettext-tt2.t b/t/misc/translator/xgettext-tt2.t new file mode 100755 index 0000000000..e2ae734914 --- /dev/null +++ b/t/misc/translator/xgettext-tt2.t @@ -0,0 +1,74 @@ +#!/usr/bin/perl + +use Modern::Perl; + +use File::Slurp; +use File::Temp qw(tempdir); +use FindBin qw($Bin); +use Locale::PO; +use Test::More tests => 36; + +my $tempdir = tempdir(CLEANUP => 1); + +write_file("$tempdir/files", "$Bin/sample.tt"); + +my $xgettext_cmd = "$Bin/../../../misc/translator/xgettext-tt2 --from-code=UTF-8 " + . "-o $tempdir/Koha.pot -f $tempdir/files"; + +system($xgettext_cmd); +my $pot = Locale::PO->load_file_asarray("$tempdir/Koha.pot"); + +my @expected = ( + { + msgid => '"hello ツ"', + }, + { + msgid => '"hello {name}"', + }, + { + msgid => '"item"', + msgid_plural => '"items"', + }, + { + msgid => '"{count} item"', + msgid_plural => '"{count} items"', + }, + { + msgid => '"hello"', + msgctxt => '"context"', + }, + { + msgid => '"hello {name}"', + msgctxt => '"context"', + }, + { + msgid => '"item"', + msgid_plural => '"items"', + msgctxt => '"context"', + }, + { + msgid => '"{count} item"', + msgid_plural => '"{count} items"', + msgctxt => '"context"', + }, + { + msgid => '"status is {status}"', + }, + { + msgid => '"active"', + }, + { + msgid => '"inactive"', + }, + { + msgid => '"Inside block"', + }, +); + +for (my $i = 0; $i < @expected; $i++) { + for my $key (qw(msgid msgid_plural msgctxt)) { + my $expected = $expected[$i]->{$key}; + my $expected_str = defined $expected ? $expected : 'not defined'; + is($pot->[$i + 1]->$key, $expected, "$i: $key is $expected_str"); + } +} diff --git a/yarn.lock b/yarn.lock index 982d18fe1c..ffecfabe02 100644 --- a/yarn.lock +++ b/yarn.lock @@ -1468,6 +1468,19 @@ gulp-cli@^2.2.0: v8flags "^3.2.0" yargs "^7.1.0" +gulp-concat-po@^1.0.0: + version "1.0.0" + resolved "https://registry.yarnpkg.com/gulp-concat-po/-/gulp-concat-po-1.0.0.tgz#2fe7b2c12e45a566238e228f63396838013770ae" + integrity sha512-hFDZrUJcpw10TW3BfptL5W2FV/aMo3M+vxz9YQV4nlMBDAi8gs9/yZYZcYMYfl5XKhjpebSef8nyruoWdlX8Hw== + dependencies: + lodash.find "^4.6.0" + lodash.merge "^4.6.2" + lodash.uniq "^4.5.0" + plugin-error "^1.0.1" + pofile "^1.1.0" + through2 "^0.6.5" + vinyl "^2.2.0" + gulp-cssnano@^2.1.2: version "2.1.3" resolved "https://registry.yarnpkg.com/gulp-cssnano/-/gulp-cssnano-2.1.3.tgz#02007e2817af09b3688482b430ad7db807aebf72" @@ -1479,6 +1492,15 @@ gulp-cssnano@^2.1.2: plugin-error "^1.0.1" vinyl-sourcemaps-apply "^0.2.1" +gulp-exec@^4.0.0: + version "4.0.0" + resolved "https://registry.yarnpkg.com/gulp-exec/-/gulp-exec-4.0.0.tgz#4b6b67be0200d620143f3198a64257b68b146bb6" + integrity sha512-A9JvTyB3P4huusd/43bTr6SDg3MqBxL9AQbLnsKSO6/91wVkHfxgeJZlgDMkqK8sMel4so8wcko4SZOeB1UCgA== + dependencies: + lodash.template "^4.4.0" + plugin-error "^1.0.1" + through2 "^3.0.1" + gulp-sass@^3.1.0: version "3.2.1" resolved "https://registry.yarnpkg.com/gulp-sass/-/gulp-sass-3.2.1.tgz#2e3688a96fd8be1c0c01340750c191b2e79fab94" @@ -2130,6 +2152,11 @@ lodash.escape@^3.0.0: dependencies: lodash._root "^3.0.0" +lodash.find@^4.6.0: + version "4.6.0" + resolved "https://registry.yarnpkg.com/lodash.find/-/lodash.find-4.6.0.tgz#cb0704d47ab71789ffa0de8b97dd926fb88b13b1" + integrity sha1-ywcE1Hq3F4n/oN6Ll92Sb7iLE7E= + lodash.isarguments@^3.0.0: version "3.1.0" resolved "https://registry.yarnpkg.com/lodash.isarguments/-/lodash.isarguments-3.1.0.tgz#2f573d85c6a24289ff00663b491c1d338ff3458a" @@ -2154,6 +2181,11 @@ lodash.memoize@^4.1.2: resolved "https://registry.yarnpkg.com/lodash.memoize/-/lodash.memoize-4.1.2.tgz#bcc6c49a42a2840ed997f323eada5ecd182e0bfe" integrity sha1-vMbEmkKihA7Zl/Mj6tpezRguC/4= +lodash.merge@^4.6.2: + version "4.6.2" + resolved "https://registry.yarnpkg.com/lodash.merge/-/lodash.merge-4.6.2.tgz#558aa53b43b661e1925a0afdfa36a9a1085fe57a" + integrity sha512-0KpjqXRVvrYyCsX1swR/XTK0va6VQkQM6MNo7PqW77ByjAhoARA8EfrP1N4+KlKj8YS0ZUCtRT/YUuhyYDujIQ== + lodash.restparam@^3.0.0: version "3.6.1" resolved "https://registry.yarnpkg.com/lodash.restparam/-/lodash.restparam-3.6.1.tgz#936a4e309ef330a7645ed4145986c85ae5b20805" @@ -2174,6 +2206,14 @@ lodash.template@^3.0.0: lodash.restparam "^3.0.0" lodash.templatesettings "^3.0.0" +lodash.template@^4.4.0: + version "4.5.0" + resolved "https://registry.yarnpkg.com/lodash.template/-/lodash.template-4.5.0.tgz#f976195cf3f347d0d5f52483569fe8031ccce8ab" + integrity sha512-84vYFxIkmidUiFxidA/KjjH9pAycqW+h980j7Fuz5qxRtO9pgB7MDFTdys1N7A5mcucRiDyEq4fusljItR1T/A== + dependencies: + lodash._reinterpolate "^3.0.0" + lodash.templatesettings "^4.0.0" + lodash.templatesettings@^3.0.0: version "3.1.1" resolved "https://registry.yarnpkg.com/lodash.templatesettings/-/lodash.templatesettings-3.1.1.tgz#fb307844753b66b9f1afa54e262c745307dba8e5" @@ -2182,6 +2222,13 @@ lodash.templatesettings@^3.0.0: lodash._reinterpolate "^3.0.0" lodash.escape "^3.0.0" +lodash.templatesettings@^4.0.0: + version "4.2.0" + resolved "https://registry.yarnpkg.com/lodash.templatesettings/-/lodash.templatesettings-4.2.0.tgz#e481310f049d3cf6d47e912ad09313b154f0fb33" + integrity sha512-stgLz+i3Aa9mZgnjr/O+v9ruKZsPsndy7qPZOchbqk2cnTU1ZaldKK+v7m54WoKIyxiuMZTKT2H81F8BeAc3ZQ== + dependencies: + lodash._reinterpolate "^3.0.0" + lodash.uniq@^4.5.0: version "4.5.0" resolved "https://registry.yarnpkg.com/lodash.uniq/-/lodash.uniq-4.5.0.tgz#d0225373aeb652adc1bc82e4945339a842754773" @@ -2284,6 +2331,11 @@ meow@^3.7.0: redent "^1.0.0" trim-newlines "^1.0.0" +merge-stream@^2.0.0: + version "2.0.0" + resolved "https://registry.yarnpkg.com/merge-stream/-/merge-stream-2.0.0.tgz#52823629a14dd00c9770fb6ad47dc6310f2c1f60" + integrity sha512-abv/qOcuPfk3URPfDzmZU1LKmuw8kT+0nIHvKrKgFrwifol/doWcdA4ZqsWQ8ENrFKkd67Mfpo/LovbIUsbt3w== + micromatch@^3.0.4, micromatch@^3.1.10, micromatch@^3.1.4: version "3.1.10" resolved "https://registry.yarnpkg.com/micromatch/-/micromatch-3.1.10.tgz#70859bc95c9840952f359a068a3fc49f9ecfac23" @@ -2753,6 +2805,11 @@ plugin-error@^1.0.1: arr-union "^3.1.0" extend-shallow "^3.0.2" +pofile@^1.1.0: + version "1.1.0" + resolved "https://registry.yarnpkg.com/pofile/-/pofile-1.1.0.tgz#9ce84bbef5043ceb4f19bdc3520d85778fad4f94" + integrity sha512-6XYcNkXWGiJ2CVXogTP7uJ6ZXQCldYLZc16wgRp8tqRaBTTyIfF+TUT3EQJPXTLAT7OTPpTAoaFdoXKfaTRU1w== + posix-character-classes@^0.1.0: version "0.1.1" resolved "https://registry.yarnpkg.com/posix-character-classes/-/posix-character-classes-0.1.1.tgz#01eac0fe3b5af71a2a6c02feabb8c1fef7e00eab" @@ -3093,6 +3150,25 @@ read-pkg@^1.0.0: normalize-package-data "^2.3.2" path-type "^1.0.0" +"readable-stream@2 || 3": + version "3.6.0" + resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-3.6.0.tgz#337bbda3adc0706bd3e024426a286d4b4b2c9198" + integrity sha512-BViHy7LKeTz4oNnkcLJ+lVSL6vpiFeX6/d3oSH8zCW7UxP2onchk+vTGB143xuFjHS3deTgkKoXXymXqymiIdA== + dependencies: + inherits "^2.0.3" + string_decoder "^1.1.1" + util-deprecate "^1.0.1" + +"readable-stream@>=1.0.33-1 <1.1.0-0": + version "1.0.34" + resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-1.0.34.tgz#125820e34bc842d2f2aaafafe4c2916ee32c157c" + integrity sha1-Elgg40vIQtLyqq+v5MKRbuMsFXw= + dependencies: + core-util-is "~1.0.0" + inherits "~2.0.1" + isarray "0.0.1" + string_decoder "~0.10.x" + readable-stream@^2.0.0, readable-stream@^2.0.2, readable-stream@^2.0.5, readable-stream@^2.1.5, readable-stream@^2.2.2, readable-stream@^2.3.3, readable-stream@^2.3.5, readable-stream@^2.3.6: version "2.3.7" resolved "https://registry.yarnpkg.com/readable-stream/-/readable-stream-2.3.7.tgz#1eca1cf711aef814c04f62252a36a62f6cb23b57" @@ -3222,9 +3298,9 @@ replace-ext@0.0.1: integrity sha1-KbvZIHinOfC8zitO5B6DeVNSKSQ= replace-ext@^1.0.0: - version "1.0.1" - resolved "https://registry.yarnpkg.com/replace-ext/-/replace-ext-1.0.1.tgz#2d6d996d04a15855d967443631dd5f77825b016a" - integrity sha512-yD5BHCe7quCgBph4rMQ+0KkIRKwWCrHDOX1p1Gp6HwjPM5kVoCdKGNhN7ydqqsX6lJEnQDKZ/tFMiEdQ1dvPEw== + version "1.0.0" + resolved "https://registry.yarnpkg.com/replace-ext/-/replace-ext-1.0.0.tgz#de63128373fcbf7c3ccfa4de5a480c45a67958eb" + integrity sha1-3mMSg3P8v3w8z6TeWkgMRaZ5WOs= replace-homedir@^1.0.0: version "1.0.0" @@ -3317,7 +3393,7 @@ rimraf@2: dependencies: glob "^7.1.3" -safe-buffer@^5.0.1, safe-buffer@^5.1.2: +safe-buffer@^5.0.1, safe-buffer@^5.1.2, safe-buffer@~5.2.0: version "5.2.0" resolved "https://registry.yarnpkg.com/safe-buffer/-/safe-buffer-5.2.0.tgz#b74daec49b1148f88c64b68d49b1e815c1f2f519" integrity sha512-fZEwUGbVl7kouZs1jCdMLdt95hdIv0ZeHg6L7qPeciMZhZ+/gdesW4wgTARkrFWEpspjEATAzUGPG8N2jJiwbg== @@ -3595,6 +3671,13 @@ string-width@^1.0.1, string-width@^1.0.2: is-fullwidth-code-point "^2.0.0" strip-ansi "^4.0.0" +string_decoder@^1.1.1: + version "1.3.0" + resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-1.3.0.tgz#42f114594a46cf1a8e30b0a84f56c78c3edac21e" + integrity sha512-hkRX8U1WjJFd8LsDJ2yQ/wWWxaopEsABU1XfkM8A+j0+85JAGppt16cr1Whg6KIbb4okU6Mql6BOj+uup/wKeA== + dependencies: + safe-buffer "~5.2.0" + string_decoder@~0.10.x: version "0.10.31" resolved "https://registry.yarnpkg.com/string_decoder/-/string_decoder-0.10.31.tgz#62e203bc41766c6c28c9fc84301dab1c5310fa94" @@ -3705,6 +3788,21 @@ through2@2.X, through2@^2.0.0, through2@^2.0.3, through2@~2.0.0: readable-stream "~2.3.6" xtend "~4.0.1" +through2@^0.6.5: + version "0.6.5" + resolved "https://registry.yarnpkg.com/through2/-/through2-0.6.5.tgz#41ab9c67b29d57209071410e1d7a7a968cd3ad48" + integrity sha1-QaucZ7KdVyCQcUEOHXp6lozTrUg= + dependencies: + readable-stream ">=1.0.33-1 <1.1.0-0" + xtend ">=4.0.0 <4.1.0-0" + +through2@^3.0.1: + version "3.0.1" + resolved "https://registry.yarnpkg.com/through2/-/through2-3.0.1.tgz#39276e713c3302edf9e388dd9c812dd3b825bd5a" + integrity sha512-M96dvTalPT3YbYLaKaCuwu+j06D/8Jfib0o/PxbVt6Amhv3dUAtW6rTV1jPgJSBG83I/e04Y6xkVdVhSRhi0ww== + dependencies: + readable-stream "2 || 3" + time-stamp@^1.0.0: version "1.1.0" resolved "https://registry.yarnpkg.com/time-stamp/-/time-stamp-1.1.0.tgz#764a5a11af50561921b133f3b44e618687e0f5c3" @@ -3889,7 +3987,7 @@ use@^3.1.0: resolved "https://registry.yarnpkg.com/use/-/use-3.1.1.tgz#d50c8cac79a19fbc20f2911f56eb973f4e10070f" integrity sha512-cwESVXlO3url9YWlFW/TA9cshCEhtu7IKJ/p5soJ/gGpj7vbvFrAY/eIioQ6Dw23KjZhYgiIo8HOs1nQ2vr/oQ== -util-deprecate@~1.0.1: +util-deprecate@^1.0.1, util-deprecate@~1.0.1: version "1.0.2" resolved "https://registry.yarnpkg.com/util-deprecate/-/util-deprecate-1.0.2.tgz#450d4dc9fa70de732762fbd2d4a28981419a0ccf" integrity sha1-RQ1Nyfpw3nMnYvvS1KKJgUGaDM8= @@ -3985,7 +4083,7 @@ vinyl@^0.5.0: clone-stats "^0.0.1" replace-ext "0.0.1" -vinyl@^2.0.0: +vinyl@^2.0.0, vinyl@^2.2.0: version "2.2.0" resolved "https://registry.yarnpkg.com/vinyl/-/vinyl-2.2.0.tgz#d85b07da96e458d25b2ffe19fece9f2caa13ed86" integrity sha512-MBH+yP0kC/GQ5GwBqrTPTzEfiiLjta7hTtvQtbxBgTeSXsmKQRQecjibMbxIXzVT3Y9KJK+drOz1/k+vsu8Nkg== @@ -4034,7 +4132,7 @@ wrappy@1: resolved "https://registry.yarnpkg.com/wrappy/-/wrappy-1.0.2.tgz#b5243d8f3ec1aa35f1364605bc0d1036e30ab69f" integrity sha1-tSQ9jz7BqjXxNkYFvA0QNuMKtp8= -xtend@~4.0.0, xtend@~4.0.1: +"xtend@>=4.0.0 <4.1.0-0", xtend@~4.0.0, xtend@~4.0.1: version "4.0.2" resolved "https://registry.yarnpkg.com/xtend/-/xtend-4.0.2.tgz#bb72779f5fa465186b1f438f674fa347fdb5db54" integrity sha512-LKYU1iAXJXUgAXn9URjiu+MWhyUXHsvfp7mcuYm9dSUKK0/CjtrUwFAxD82/mCWbtLsGjFIad0wIsod4zrTAEQ== -- 2.20.1