Bug 24211 - Compress/uncompress translation files
Summary: Compress/uncompress translation files
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Bernardo Gonzalez Kriegel
QA Contact: Testopia
URL:
Keywords: release-notes-needed
Depends on:
Blocks:
 
Reported: 2019-12-11 00:00 UTC by Bernardo Gonzalez Kriegel
Modified: 2020-03-31 22:11 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.05.00


Attachments
Bug 24211: Compress/uncompress translation files (5.33 KB, patch)
2019-12-11 00:18 UTC, Bernardo Gonzalez Kriegel
Details | Diff | Splinter Review
Bug 24211: Compress/uncompress translation files (5.39 KB, patch)
2020-01-04 20:00 UTC, David Nind
Details | Diff | Splinter Review
Bug 24211: Compress/uncompress translation files (5.44 KB, patch)
2020-02-07 19:03 UTC, Bouzid Fergani
Details | Diff | Splinter Review
Bug 24211: Compress/uncompress translation files (5.53 KB, patch)
2020-02-21 13:17 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bernardo Gonzalez Kriegel 2019-12-11 00:00:04 UTC
Translation files takes some space in an installed system, ~350 MB. It's not too much with current drives, but it's a space that can be saved on container images.

Compressing them it's easy and can be made transparently, using only ~50 MB.

Perhaps this enhancement could be used when generating packages.
Comment 1 Bernardo Gonzalez Kriegel 2019-12-11 00:18:50 UTC
Created attachment 96174 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)
Comment 2 Martin Renvoize 2020-01-02 14:35:56 UTC
Nice idea :)
Comment 3 David Nind 2020-01-04 20:00:14 UTC
Created attachment 96814 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 Bouzid Fergani 2020-02-07 19:03:41 UTC
Created attachment 98584 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Comment 5 Jonathan Druart 2020-02-12 14:30:21 UTC
Bernardo, could you explain the different use cases here?

Is the idea to commit the gz files?
Comment 6 Bernardo Gonzalez Kriegel 2020-02-12 15:41:16 UTC
(In reply to Jonathan Druart from comment #5)
> Bernardo, could you explain the different use cases here?

I see the following:
1) Compression before release (deb/tar.gz)
In this case the installed system will use less space

2) Smaller docker images, perhaps using compressed files from deb/tar

> 
> Is the idea to commit the gz files?

I don't think so. I don't know much about git, but I think it would be worse for the size of the repository.
Comment 7 Jonathan Druart 2020-02-21 13:17:47 UTC
Created attachment 99375 [details] [review]
Bug 24211: Compress/uncompress translation files

This patch adds the ability to compress/uncompress
translations files.

On update/install the files are uncompressed first

The only gain is to use less space.

To test:
1) Apply the patch
2) Go to misc/translator
3) Try it
   $ ./translate compress fr-FR (check po/*.gz)
   $ ./translate uncompress fr-FR ( check normal files)
4) Try again with verbose mode
   $ ./translate compress fr-FR -v (list compressed files)
   $ ./translate uncompress fr-FR -v (list uncompressed files)
5) Try update compressed files
   $ ./translate compress fr-FR
   $ ./translate update fr-FR (result is uncompressed)
6) Try install compressed files
   $ ./translate compress fr-FR
   $ ./translate install fr-FR
7) Try compress all
   $ ./translate compress  (add '-v' for verbose output)
8) Try uncompress all
   $ ./translate uncompress  (add '-v' for verbose output)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>

Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Comment 8 Jonathan Druart 2020-02-21 13:18:32 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #6)
> (In reply to Jonathan Druart from comment #5)
> 2) Smaller docker images, perhaps using compressed files from deb/tar
> 
> > 
> > Is the idea to commit the gz files?
> 
> I don't think so. I don't know much about git, but I think it would be worse
> for the size of the repository.

We must not, that's why I was asking :)
Comment 9 Julian Maurice 2020-02-21 13:41:36 UTC
Have you tried to compile them (with msgfmt) instead of compressing them ?
Maybe .mo files can be read by the translator tool without being uncompressed on disk first.
Comment 10 Bernardo Gonzalez Kriegel 2020-02-21 14:07:22 UTC
(In reply to Julian Maurice from comment #9)
> Have you tried to compile them (with msgfmt) instead of compressing them ?
> Maybe .mo files can be read by the translator tool without being
> uncompressed on disk first.

Current translation process does not use .mo files, we only use .po files as a bag of matching strings (msgid -> msgsrt) to do search and replace inspecting tt, xml and js files.

Besides, .mo binary files are not much smaller than .po files :)
Comment 11 Julian Maurice 2020-02-21 14:51:22 UTC
(In reply to Bernardo Gonzalez Kriegel from comment #10)
> Current translation process does not use .mo files, we only use .po files as
> a bag of matching strings (msgid -> msgsrt) to do search and replace
> inspecting tt, xml and js files.

I know that. What I suggested was to use .mo files as bag of strings too. But that may require additional dependencies, so it may not be worth it, especially if:

> .mo binary files are not much smaller than .po files :)

I just did a small test:

msgfmt -o fr-FR-staff-prog.mo fr-FR-staff-prog.po
gzip -k fr-FR-staff-prog.po
du -h fr-FR-staff-prog*

824K    fr-FR-staff-prog.mo
2,1M    fr-FR-staff-prog.po
392K    fr-FR-staff-prog.po.gz

So, .mo files are smaller than .po files, but not as small as gzipped .po
Which is kind of disappointing :/

Anyway, I just wanted to know if you had considered this approach. Thanks for answering.
Comment 12 Martin Renvoize 2020-02-24 13:21:58 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 13 Joy Nelson 2020-03-31 22:11:27 UTC
enhancement not backported to 19.11.x branch