Bug 35079 - Add option to gulp tasks po:update and po:create to control if POT should be built
Summary: Add option to gulp tasks po:update and po:create to control if POT should be ...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Julian Maurice
QA Contact: Testopia
URL:
Keywords:
Depends on: 35024
Blocks: 35332 35103
  Show dependency treegraph
 
Reported: 2023-10-17 14:42 UTC by Julian Maurice
Modified: 2023-11-14 13:26 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.11.00,23.05.6,22.11.12,22.05.17,21.11.26


Attachments
Bug 35079: Rebuild POT files only if necessary or asked explicitely (4.78 KB, patch)
2023-10-17 14:43 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 35079: Replace --force-extract by --generate-pot={always,auto,never} (2.52 KB, patch)
2023-10-17 15:17 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 35079: Rebuild POT files only if necessary or asked explicitely (4.85 KB, patch)
2023-10-18 11:07 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 35079: Replace --force-extract by --generate-pot={always,auto,never} (2.60 KB, patch)
2023-10-18 11:07 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2023-10-17 14:42:38 UTC

    
Comment 1 Julian Maurice 2023-10-17 14:43:15 UTC
Created attachment 157253 [details] [review]
Bug 35079: Rebuild POT files only if necessary or asked explicitely

By default, gulp tasks po:create and po:update won't rebuild the POT
files if they already exist.
Both tasks gained a new option --force-extract to rebuild them
unconditionally

This makes it possible to create/update PO files for multiple languages
sequentially without rebuilding the POT file for each language.
For instance:

    gulp po:update --lang fr-FR
    # do something with the PO files
    gulp po:update --lang en-GB
    # ...
Comment 2 Julian Maurice 2023-10-17 15:17:28 UTC
Created attachment 157257 [details] [review]
Bug 35079: Replace --force-extract by --generate-pot={always,auto,never}

This restores the original behaviour of always building the POT file
in order to not break existing workflows
Option --force-extract is deleted in favor of a new option
--generate-pot that can have 3 values:

* always: always build the POT file. This is the default value
* auto: build the POT file only if it does not exist
* never: never build the POT file
Comment 3 Jonathan Druart 2023-10-18 10:14:55 UTC
time gulp po:update --lang az-AZ --lang prs --lang mon --lang tl-PH --lang te --lang ms-MY --lang ur-Arab --lang rw-RW --lang pbr --lang he-Hebr --lang ku-Arab --lan
g de-CH --lang cy-GB --lang my --lang nl-BE --lang is-IS --lang da-DK --lang ia --lang am-Ethi --lang fr-FR --lang sw-KE --lang ne-NE --lang ru-RU --lang iq-CA --lang si --lang fo-FO --lang fa-Arab --lang bn-IN --
lang uk-UA --lang hu-HU --lang tk --lang ar-Arab --lang as-IN --lang nn-NO --lang lo-Laoo --lang ka --lang ml --lang et --lang sq-AL --lang uz --lang iu-CA --lang sk-SK --lang km-KH --lang tr-TR --lang en-NZ --lan
g nl-NL --lang tvl --lang ta-LK --lang hi --lang hyw-Armn --lang pt-PT --lang vi-VN --lang ko-Kore-KP --lang lv --lang it-IT --lang ca-ES --lang fr-CA --lang hy-Armn --lang mr --lang pt-BR --lang mi-NZ --lang sl-S
I --lang sv-SE --lang dz --lang th-TH --lang pl-PL --lang zh-Hans-CN --lang oc --lang de-DE --lang es-ES --lang cs-CZ --lang gd --lang nb-NO --lang gl --lang ja-Jpan-JP --lang sr-Cyrl --lang id-ID --lang ga --lang
 ta --lang bg-Cyrl --lang en-GB --lang tet --lang hr-HR --lang el-GR --lang eo --lang ro-RO --lang fi-FI --lang sd-PK --lang eu --lang zh-Hant-TW --lang kn-Knda --lang be-BY

real    0m57.700s



time ( gulp po:extract && gulp po:update --generate-pot never --lang az-AZ --lang prs --lang mon --lang tl-PH --lang te --lang ms-MY --lang ur-Arab --lang rw-RW --l
ang pbr --lang he-Hebr --lang ku-Arab --lang de-CH --lang cy-GB --lang my --lang nl-BE --lang is-IS --lang da-DK --lang ia --lang am-Ethi --lang fr-FR --lang sw-KE --lang ne-NE --lang ru-RU --lang iq-CA --lang si 
--lang fo-FO --lang fa-Arab --lang bn-IN --lang uk-UA --lang hu-HU --lang tk --lang ar-Arab --lang as-IN --lang nn-NO --lang lo-Laoo --lang ka --lang ml --lang et --lang sq-AL --lang uz --lang iu-CA --lang sk-SK -
-lang km-KH --lang tr-TR --lang en-NZ --lang nl-NL --lang tvl --lang ta-LK --lang hi --lang hyw-Armn --lang pt-PT --lang vi-VN --lang ko-Kore-KP --lang lv --lang it-IT --lang ca-ES --lang fr-CA --lang hy-Armn --la
ng mr --lang pt-BR --lang mi-NZ --lang sl-SI --lang sv-SE --lang dz --lang th-TH --lang pl-PL --lang zh-Hans-CN --lang oc --lang de-DE --lang es-ES --lang cs-CZ --lang gd --lang nb-NO --lang gl --lang ja-Jpan-JP -
-lang sr-Cyrl --lang id-ID --lang ga --lang ta --lang bg-Cyrl --lang en-GB --lang tet --lang hr-HR --lang el-GR --lang eo --lang ro-RO --lang fi-FI --lang sd-PK --lang eu --lang zh-Hant-TW --lang kn-Knda --lang be
-BY)

real    0m34.250s
Comment 4 Jonathan Druart 2023-10-18 10:15:23 UTC
(there is a bug when using gulp po:update without --lang)
Comment 5 Jonathan Druart 2023-10-18 11:07:09 UTC
Created attachment 157304 [details] [review]
Bug 35079: Rebuild POT files only if necessary or asked explicitely

By default, gulp tasks po:create and po:update won't rebuild the POT
files if they already exist.
Both tasks gained a new option --force-extract to rebuild them
unconditionally

This makes it possible to create/update PO files for multiple languages
sequentially without rebuilding the POT file for each language.
For instance:

    gulp po:update --lang fr-FR
    # do something with the PO files
    gulp po:update --lang en-GB
    # ...

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 6 Jonathan Druart 2023-10-18 11:07:12 UTC
Created attachment 157305 [details] [review]
Bug 35079: Replace --force-extract by --generate-pot={always,auto,never}

This restores the original behaviour of always building the POT file
in order to not break existing workflows
Option --force-extract is deleted in favor of a new option
--generate-pot that can have 3 values:

* always: always build the POT file. This is the default value
* auto: build the POT file only if it does not exist
* never: never build the POT file

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 7 Jonathan Druart 2023-10-19 07:14:52 UTC
Note that if you see:
  Error: Cannot find module 'stream/promises'

It means that you need a more recent version of node.

I was using v14 on my host (which I don't usually use for that), and it needs at least 15.

We have v18 in ktd.
Comment 8 Julian Maurice 2023-10-19 07:34:02 UTC
If compatibility with older node.js versions is wanted, it's possible to use the non-promise version. But v14 and v16 are EOL now, so they shouldn't be used anyway.
Comment 9 Jonathan Druart 2023-10-27 11:05:57 UTC
Logging the following here.

There is a bug when you run `gulp po:update` without --lang (or with all langs passed).

Error: Command failed: msgmerge --backup=off --no-wrap --quiet -F --update misc/translator/po/en-NZ-messages.po misc/translator/Koha-messages.pot                                                         
misc/translator/Koha-messages.pot:11:35: syntax error                                                                                                                                                                
misc/translator/Koha-messages.pot:11: keyword "n" unknown                                                                                                                                                            
misc/translator/Koha-messages.pot:12: end-of-line within string      


There is an extra quote at the end of line 11 ("\n" instead of \n").

Doing a po:extract then a po:update seems to fix the problem.
Comment 10 Jonathan Druart 2023-10-27 11:07:06 UTC
(In reply to Jonathan Druart from comment #9)
> There is an extra quote at the end of line 11 ("\n" instead of \n").

11 "Content-Transfer-Encoding: 8bit"\n"
Comment 11 Jonathan Druart 2023-10-31 09:18:39 UTC
Passing QA, I need it ASAP.
Comment 12 Tomás Cohen Arazi 2023-11-01 20:26:56 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 13 Fridolin Somers 2023-11-08 08:45:55 UTC
Depends on Bug 35024 not yet in 23.05.x
Comment 14 Marcel de Rooy 2023-11-13 10:54:26 UTC
(In reply to Jonathan Druart from comment #7)
> Note that if you see:
>   Error: Cannot find module 'stream/promises'
> 
> It means that you need a more recent version of node.
> 
> I was using v14 on my host (which I don't usually use for that), and it
> needs at least 15.
> 
> We have v18 in ktd.

I have version 18
nodejs/nodistro,now 18.18.2-1nodesource1 amd64 [installed]

But I still get this stream/promises error after build-resources tries to run gulp build && gulp build --view opac

Error: Cannot find module 'stream/promises'
Require stack:
- /usr/share/koha/gulpfile.js
- /usr/share/koha/node_modules/gulp-cli/lib/shared/require-or-import.js
- /usr/share/koha/node_modules/gulp-cli/lib/versioned/^4.0.0/index.js
- /usr/share/koha/node_modules/gulp-cli/index.js
- /usr/share/koha/node_modules/gulp/bin/gulp.js
    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:931:15)
    at Function.Module._load (internal/modules/cjs/loader.js:774:27)
    at Module.require (internal/modules/cjs/loader.js:1003:19)
    at require (internal/modules/cjs/helpers.js:107:18)
    at Object.<anonymous> (/usr/share/koha/gulpfile.js:11:16)
    at Module._compile (internal/modules/cjs/loader.js:1114:14)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:1143:10)
    at Module.load (internal/modules/cjs/loader.js:979:32)
    at Function.Module._load (internal/modules/cjs/loader.js:819:12)
    at Module.require (internal/modules/cjs/loader.js:1003:19) {
  code: 'MODULE_NOT_FOUND',
  requireStack: [
    '/usr/share/koha/gulpfile.js',
    '/usr/share/koha/node_modules/gulp-cli/lib/shared/require-or-import.js',
    '/usr/share/koha/node_modules/gulp-cli/lib/versioned/^4.0.0/index.js',
    '/usr/share/koha/node_modules/gulp-cli/index.js',
    '/usr/share/koha/node_modules/gulp/bin/gulp.js'
  ]
}
Comment 15 Julian Maurice 2023-11-13 11:05:36 UTC
(In reply to Marcel de Rooy from comment #14)
> I have version 18
> nodejs/nodistro,now 18.18.2-1nodesource1 amd64 [installed]

Does `node --version` return 18 ? Maybe you have an older version installed in /usr/local are something like that.
Comment 16 Marcel de Rooy 2023-11-13 12:43:38 UTC
(In reply to Julian Maurice from comment #15)
> (In reply to Marcel de Rooy from comment #14)
> > I have version 18
> > nodejs/nodistro,now 18.18.2-1nodesource1 amd64 [installed]
> 
> Does `node --version` return 18 ? Maybe you have an older version installed
> in /usr/local are something like that.

Thats correct (somehow):
node --version
v14.21.3
Comment 17 Marcel de Rooy 2023-11-13 12:47:06 UTC
Yeah, nice:

/usr/bin/node --version
v18.18.2
Comment 18 Marcel de Rooy 2023-11-13 13:30:34 UTC
Just adding a few lines for anyone who reads this later. Resolved situation by:

Using NODE_MAJOR=18

# npm / nodejs
    mkdir -p /etc/apt/keyrings; \
    curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg; \
    echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee /etc/apt/sources.list.d/nodesource.list; \
    apt-get update; \
    apt-get -yq install nodejs; \
    npm install -g yarn; \

Note that you cannot apt-get install nodejs npm. This gives a conflict. npm is part of the above install of nodejs.
Comment 19 Jonathan Druart 2023-11-14 07:51:34 UTC
(In reply to Marcel de Rooy from comment #18)
> Just adding a few lines for anyone who reads this later. Resolved situation
> by:
> 
> Using NODE_MAJOR=18
> 
> # npm / nodejs
>     mkdir -p /etc/apt/keyrings; \
>     curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key |
> gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg; \
>     echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg]
> https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | tee
> /etc/apt/sources.list.d/nodesource.list; \
>     apt-get update; \
>     apt-get -yq install nodejs; \
>     npm install -g yarn; \
> 
> Note that you cannot apt-get install nodejs npm. This gives a conflict. npm
> is part of the above install of nodejs.

Yes, see also https://gitlab.com/koha-community/koha-testing-docker/-/issues/404
Comment 20 Jonathan Druart 2023-11-14 11:46:00 UTC
(In reply to Jonathan Druart from comment #9)
> Logging the following here.
> 
> There is a bug when you run `gulp po:update` without --lang (or with all
> langs passed).
> 
> Error: Command failed: msgmerge --backup=off --no-wrap --quiet -F --update
> misc/translator/po/en-NZ-messages.po misc/translator/Koha-messages.pot      
> 
> misc/translator/Koha-messages.pot:11:35: syntax error                       
> 
> misc/translator/Koha-messages.pot:11: keyword "n" unknown                   
> 
> misc/translator/Koha-messages.pot:12: end-of-line within string      
> 
> 
> There is an extra quote at the end of line 11 ("\n" instead of \n").
> 
> Doing a po:extract then a po:update seems to fix the problem.

See bug 35332.