Bug 24262 - Translate installer data in YAML format
Summary: Translate installer data in YAML format
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Jonathan Druart
URL:
Keywords:
Depends on: 13897
Blocks: 24712 24583 24584 24593 24594 32356
  Show dependency treegraph
 
Reported: 2019-12-18 12:37 UTC by Bernardo Gonzalez Kriegel
Modified: 2022-12-04 12:33 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00


Attachments
Bug 24262: Translate installer data in YAML format (13.27 KB, patch)
2019-12-20 19:08 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24262: Translate installer data in YAML format (13.21 KB, patch)
2019-12-20 19:17 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24262: Translate installer data in YAML format (14.03 KB, patch)
2019-12-23 02:06 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24262: Translate installer data in YAML format (14.19 KB, patch)
2019-12-25 13:29 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24262: Translate installer data in YAML format (17.97 KB, patch)
2020-02-05 15:22 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24262: Translate installer data in YAML format (18.09 KB, patch)
2020-02-21 16:25 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 24262: (followup) admit null values on labels (1.71 KB, patch)
2020-03-05 13:54 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24262: Translate installer data in YAML format (18.25 KB, patch)
2020-03-09 12:18 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24262: (followup) admit null values on labels (1.78 KB, patch)
2020-03-09 12:18 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bernardo Gonzalez Kriegel 2019-12-18 12:37:00 UTC
This complements Bug 13897, adding ability to translate YAML files in installer dirs.
Comment 1 Bernardo Gonzalez Kriegel 2019-12-20 19:08:53 UTC
Created attachment 96538 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files) and Bug 18688 (for warnings)

To test:
1) Apply patches from Bug 13897 and Bug 18688.
2) Apply this patch
3) Go to misc/translation
4) Create new translation files for a language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR )
   Install dir is ignored and not changed.
Comment 2 Bernardo Gonzalez Kriegel 2019-12-20 19:17:16 UTC
Created attachment 96540 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files) and Bug 18688 (for warnings)

To test:
1) Apply patches from Bug 13897 and Bug 18688.
2) Apply this patch
3) Go to misc/translation
4) Create new translation files for a language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR )
   Install dir is ignored and not changed.
Comment 3 Bernardo Gonzalez Kriegel 2019-12-23 02:06:19 UTC
Created attachment 96593 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files) and Bug 18688 (for warnings)

To test:
1) Apply patches from Bug 13897 and Bug 18688.
2) Apply this patch
3) Go to misc/translation
4) Create new translation files for a language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR )
   Install dir is ignored and not changed.
Comment 4 Jonathan Druart 2019-12-23 14:20:06 UTC
Comment on attachment 96593 [details] [review]
Bug 24262: Translate installer data in YAML format

Review of attachment 96593 [details] [review]:
-----------------------------------------------------------------

::: misc/translator/LangInstaller.pm
@@ +937,4 @@
>      $self->install_prefs();
>      $self->install_messages();
>      $self->remove_pot();
> +    $self->install_installer();

Should not it be placed after remove_pot?

@@ +971,4 @@
>      $self->create_prefs();
>      $self->create_messages();
>      $self->remove_pot();
> +    $self->create_installer();

Same here.
Comment 5 Bernardo Gonzalez Kriegel 2019-12-23 15:00:58 UTC
Hola Jonathan!

(In reply to Jonathan Druart from comment #4)
> ...
> >      $self->install_prefs();
> >      $self->install_messages();
> >      $self->remove_pot();
> > +    $self->install_installer();
> 
> Should not it be placed after remove_pot?

The order is not relevant in this case, it can be at the beginning, end or middle. The new functions works independently of the old ones, with different target dirs.
Comment 6 Bernardo Gonzalez Kriegel 2019-12-25 13:29:51 UTC
Created attachment 96626 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files)

To test:
1) Apply patches from Bug 13897
2) Apply this patch
3) Go to misc/translation
4) Create translation files for a NEW language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values table for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR ), eg.
   ./translate install de-DE

   Install dir is detected and not changed, a note is printed.
Comment 7 Bernardo Gonzalez Kriegel 2020-02-05 15:22:18 UTC
Created attachment 98506 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files)

NOTE: updated version adding ability to process multiline
fields, discard small ( < 2) strings, and discard strings
with pure html, TT or punctuation.

To test:
1) Apply patches from Bug 13897
2) Apply this patch
3) Go to misc/translation
4) Create translation files for a NEW language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values table for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR ), eg.
   ./translate install de-DE

   Install dir is detected and not changed, a note is printed.
Comment 8 Martin Renvoize 2020-02-21 16:25:29 UTC
Created attachment 99396 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files)

NOTE: updated version adding ability to process multiline
fields, discard small ( < 2) strings, and discard strings
with pure html, TT or punctuation.

To test:
1) Apply patches from Bug 13897
2) Apply this patch
3) Go to misc/translation
4) Create translation files for a NEW language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values table for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR ), eg.
   ./translate install de-DE

   Install dir is detected and not changed, a note is printed.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize 2020-02-21 16:26:52 UTC
Followed the test plan and all looks good to me.. Signing off
Comment 10 Martin Renvoize 2020-02-22 09:09:38 UTC
OK.. I'm loving this work..

I just had a moment of inspiration though.. what about upgrades?

Currently, we add to both an atomicupdate file (which gets put into updatedatabase) and add nearly the same code into various .SQL files (now becoming .yaml files) for install time.  I'm now wondering if as a further bug we could add a function into updatedatabase to walk through the tree of yaml files to install translated missing default data at the end of any run of updatedatabase and save ourselves from having to write the atomicupdate and the yaml file for such DB changes?
Comment 11 Katrin Fischer 2020-02-23 09:35:56 UTC
(In reply to Martin Renvoize from comment #10)
> OK.. I'm loving this work..
> 
> I just had a moment of inspiration though.. what about upgrades?
> 
> Currently, we add to both an atomicupdate file (which gets put into
> updatedatabase) and add nearly the same code into various .SQL files (now
> becoming .yaml files) for install time.  I'm now wondering if as a further
> bug we could add a function into updatedatabase to walk through the tree of
> yaml files to install translated missing default data at the end of any run
> of updatedatabase and save ourselves from having to write the atomicupdate
> and the yaml file for such DB changes?

I think this might get a little too complicated. For example libraries might have removed sample data on purpose, changed or reconfigured it. It would be hard to differentiate. We will want to add a new notice on update, but should respect if a library deleted it etc. (hope I understood correctly). So I think having different mechanisms (triggered for new and for a specific update) would be better.
Comment 12 Bernardo Gonzalez Kriegel 2020-03-05 13:54:24 UTC
Created attachment 100194 [details] [review]
Bug 24262: (followup) admit null values on labels

This patch adds handling of null values on translatable
labels.
Comment 13 Jonathan Druart 2020-03-09 12:18:50 UTC
Created attachment 100344 [details] [review]
Bug 24262: Translate installer data in YAML format

This patch adds the ability to:
1) Create new translation files from yaml installer files
2) Create installer directory for a given language

It will not create a installer directory if it already exists.

New (possible) translation files:
  xx-YY-installer.po
  xx-YY-installer-MARC21.po
  xx-YY-installer-UNIMARC.po

Needs Bug 13897 (for yaml files)

NOTE: updated version adding ability to process multiline
fields, discard small ( < 2) strings, and discard strings
with pure html, TT or punctuation.

To test:
1) Apply patches from Bug 13897
2) Apply this patch
3) Go to misc/translation
4) Create translation files for a NEW language
   $ ./translate create xx-YY

   check new file 'xx-YY-installer.po'

5) Copy ../../installer/data/mysql/en/optional/auth_val.yml
   into ../../installer/data/mysql/en/marcflavour/marc21/mandatory/
   and ../../installer/data/mysql/en/marcflavour/unimarc/mandatory/

   remove po/xx-YY*, then repeat creation

   check new files 'xx-YY-installer.po', 'xx-YY-installer-MARC21.po'
   and 'xx-YY-installer-UNIMARC.po'

   remove all new files

6) Create for xx-YY again and try update
   $ ./translate create xx-YY
   edit ../../installer/data/mysql/en/optional/auth_val.yml
   and change one char in one of the translatable values,
   also edit 'xx-YY-installer.po', translate the same
   string (in msgstr).

   Do an update
   $ ./translate update xx-YY

   check in 'xx-YY-installer.po' a fuzzy value for the
   translated value and the preservation of the translation
   Fix the translation, or add a new one.

7) Create an install dir for xx-YY
   ./translate install xx-YY

    Check new dir '../../installer/data/mysql/xx-YY/'
    Check files on it
    $ tree ../../installer/data/mysql/xx-YY/
    and compare with ../../installer/data/mysql/en/

    All installation files must be present

8) Try a new Koha install using this language in the
   usual way.
   Check in authorised_values table for the translated string.

9) Try create an install dir for an existing language
   (eg. es-ES, de-DE or fr-FR ), eg.
   ./translate install de-DE

   Install dir is detected and not changed, a note is printed.

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 14 Jonathan Druart 2020-03-09 12:18:54 UTC
Created attachment 100345 [details] [review]
Bug 24262: (followup) admit null values on labels

This patch adds handling of null values on translatable
labels.

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 15 Martin Renvoize 2020-03-09 14:12:28 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 16 Joy Nelson 2020-04-01 22:35:09 UTC
enhancement not backported to 19.11.x
Comment 17 Jonathan Druart 2022-11-29 09:12:34 UTC
Bernardo, do yo remember why you added that?

> It will not create a installer directory if it already exists.

379     if ( -d $langdir ) {
380         say "$self->{lang} installer dir $langdir already exists.\nDelete it if you want to recreate it." if $self->{verbose};
381         return;
382     }


On upgrade we want to update the directory.
Comment 18 Katrin Fischer 2022-12-04 12:33:25 UTC
(In reply to Jonathan Druart from comment #17)
> Bernardo, do yo remember why you added that?
> 
> > It will not create a installer directory if it already exists.
> 
> 379     if ( -d $langdir ) {
> 380         say "$self->{lang} installer dir $langdir already
> exists.\nDelete it if you want to recreate it." if $self->{verbose};
> 381         return;
> 382     }
> 
> 
> On upgrade we want to update the directory.

I agree, we want/need to update the files on update. I don't think we could/should handle this differently than other files that are overwritten.