Bug 24211 - Compress/uncompress translation files
Summary: Compress/uncompress translation files
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-12-11 00:00 UTC by Bernardo Gonzalez Kriegel
Modified: 2020-12-15 01:38 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Text to go in the release notes:
Allows to compress and uncompress .po files used for translations using: ./translate compress fr-FR ./translate uncompress fr-FR
Version(s) released in:
20.05.00


Attachments
Bug 24211: Compress/uncompress translation files (5.33 KB, patch)
2019-12-11 00:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24211: Compress/uncompress translation files (5.39 KB, patch)
2020-01-04 20:00 UTC, David Nind
Details | Diff | Splinter Review
Bug 24211: Compress/uncompress translation files (5.44 KB, patch)
2020-02-07 19:03 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 24211: Compress/uncompress translation files (5.53 KB, patch)
2020-02-21 13:17 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bernardo Gonzalez Kriegel 2019-12-11 00:00:04 UTC
Translation files takes some space in an installed system, ~350 MB. It's not too much with current drives, but it's a space that can be saved on container images.

Compressing them it's easy and can be made transparently, using only ~50 MB.

Perhaps this enhancement could be used when generating packages.
Comment 1 Bernardo Gonzalez Kriegel 2019-12-11 00:18:50 UTC
Created attachment 96174 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)
Comment 2 Martin Renvoize 2020-01-02 14:35:56 UTC
Nice idea :)
Comment 3 David Nind 2020-01-04 20:00:14 UTC
Created attachment 96814 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 Bouzid Fergani 2020-02-07 19:03:41 UTC
Created attachment 98584 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 5 Jonathan Druart 2020-02-12 14:30:21 UTC
Bernardo, could you explain the different use cases here?

Is the idea to commit the gz files?
Comment 6 Bernardo Gonzalez Kriegel 2020-02-12 15:41:16 UTC
(In reply to Jonathan Druart from comment #5)
> Bernardo, could you explain the different use cases here?

I see the following:
1) Compression before release (deb/tar.gz)
In this case the installed system will use less space

2) Smaller docker images, perhaps using compressed files from deb/tar

> 
> Is the idea to commit the gz files?

I don't think so. I don't know much about git, but I think it would be worse for the size of the repository.
Comment 7 Jonathan Druart 2020-02-21 13:17:47 UTC
Created attachment 99375 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 8 Jonathan Druart 2020-02-21 13:18:32 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #6)
> (In reply to Jonathan Druart from comment #5)
> 2) Smaller docker images, perhaps using compressed files from deb/tar
> 
> > 
> > Is the idea to commit the gz files?
> 
> I don't think so. I don't know much about git, but I think it would be worse
> for the size of the repository.

We must not, that's why I was asking :)
Comment 9 Julian Maurice 2020-02-21 13:41:36 UTC
Have you tried to compile them (with msgfmt) instead of compressing them ?
Maybe .mo files can be read by the translator tool without being uncompressed on disk first.
Comment 10 Bernardo Gonzalez Kriegel 2020-02-21 14:07:22 UTC
(In reply to Julian Maurice from comment #9)
> Have you tried to compile them (with msgfmt) instead of compressing them ?
> Maybe .mo files can be read by the translator tool without being
> uncompressed on disk first.

Current translation process does not use .mo files, we only use .po files as a bag of matching strings (msgid -> msgsrt) to do search and replace inspecting tt, xml and js files.

Besides, .mo binary files are not much smaller than .po files :)
Comment 11 Julian Maurice 2020-02-21 14:51:22 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #10)
> Current translation process does not use .mo files, we only use .po files as
> a bag of matching strings (msgid -> msgsrt) to do search and replace
> inspecting tt, xml and js files.

I know that. What I suggested was to use .mo files as bag of strings too. But that may require additional dependencies, so it may not be worth it, especially if:

> .mo binary files are not much smaller than .po files :)

I just did a small test:

msgfmt -o fr-FR-staff-prog.mo fr-FR-staff-prog.po
gzip -k fr-FR-staff-prog.po
du -h fr-FR-staff-prog*

824K    fr-FR-staff-prog.mo
2,1M    fr-FR-staff-prog.po
392K    fr-FR-staff-prog.po.gz

So, .mo files are smaller than .po files, but not as small as gzipped .po
Which is kind of disappointing :/

Anyway, I just wanted to know if you had considered this approach. Thanks for answering.
Comment 12 Martin Renvoize 2020-02-24 13:21:58 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 13 Joy Nelson 2020-03-31 22:11:27 UTC
enhancement not backported to 19.11.x branch
Comment 14 Katrin Fischer 2020-11-16 21:52:54 UTC
I tried to describe this one, but I am not compeletely sure how it fits in with the other new translation features.
Comment 15 David Cook 2020-12-15 01:19:38 UTC
Interesting idea!
Comment 16 David Cook 2020-12-15 01:33:56 UTC
I've opened Bug 27236 for the idea of packaging compressed PO files separately  in a koha-i18n package...
Comment 17 David Cook 2020-12-15 01:38:39 UTC
English trivia:

I was thinking that uncompress wasn't a word and that it should be decompress. However, it appears that uncompress is a word! Decompress dates back to 1905 and uncompress dates back to 1987. It even looks like uncompress actually came into usage because of computer compression. (My spellchecker doesn't like uncompress, but it appears to be wrong.)

https://www.merriam-webster.com/dictionary/decompress
https://www.merriam-webster.com/dictionary/uncompress