Translation files takes some space in an installed system, ~350 MB. It's not too much with current drives, but it's a space that can be saved on container images. Compressing them it's easy and can be made transparently, using only ~50 MB. Perhaps this enhancement could be used when generating packages.
Created attachment 96174 [details] [review] Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output)
Nice idea :)
Created attachment 96814 [details] [review] Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) Signed-off-by: David Nind <david@davidnind.com>
Created attachment 98584 [details] [review] Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com>
Bernardo, could you explain the different use cases here? Is the idea to commit the gz files?
(In reply to Jonathan Druart from comment #5) > Bernardo, could you explain the different use cases here? I see the following: 1) Compression before release (deb/tar.gz) In this case the installed system will use less space 2) Smaller docker images, perhaps using compressed files from deb/tar > > Is the idea to commit the gz files? I don't think so. I don't know much about git, but I think it would be worse for the size of the repository.
Created attachment 99375 [details] [review] Bug 24211: Compress/uncompress translation files This patch adds the ability to compress/uncompress translations files. On update/install the files are uncompressed first The only gain is to use less space. To test: 1) Apply the patch 2) Go to misc/translator 3) Try it $ ./translate compress fr-FR (check po/*.gz) $ ./translate uncompress fr-FR ( check normal files) 4) Try again with verbose mode $ ./translate compress fr-FR -v (list compressed files) $ ./translate uncompress fr-FR -v (list uncompressed files) 5) Try update compressed files $ ./translate compress fr-FR $ ./translate update fr-FR (result is uncompressed) 6) Try install compressed files $ ./translate compress fr-FR $ ./translate install fr-FR 7) Try compress all $ ./translate compress (add '-v' for verbose output) 8) Try uncompress all $ ./translate uncompress (add '-v' for verbose output) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Bouzid Fergani <bouzid.fergani@inlibro.com> Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
(In reply to Bernardo Gonzalez Kriegel from comment #6) > (In reply to Jonathan Druart from comment #5) > 2) Smaller docker images, perhaps using compressed files from deb/tar > > > > > Is the idea to commit the gz files? > > I don't think so. I don't know much about git, but I think it would be worse > for the size of the repository. We must not, that's why I was asking :)
Have you tried to compile them (with msgfmt) instead of compressing them ? Maybe .mo files can be read by the translator tool without being uncompressed on disk first.
(In reply to Julian Maurice from comment #9) > Have you tried to compile them (with msgfmt) instead of compressing them ? > Maybe .mo files can be read by the translator tool without being > uncompressed on disk first. Current translation process does not use .mo files, we only use .po files as a bag of matching strings (msgid -> msgsrt) to do search and replace inspecting tt, xml and js files. Besides, .mo binary files are not much smaller than .po files :)
(In reply to Bernardo Gonzalez Kriegel from comment #10) > Current translation process does not use .mo files, we only use .po files as > a bag of matching strings (msgid -> msgsrt) to do search and replace > inspecting tt, xml and js files. I know that. What I suggested was to use .mo files as bag of strings too. But that may require additional dependencies, so it may not be worth it, especially if: > .mo binary files are not much smaller than .po files :) I just did a small test: msgfmt -o fr-FR-staff-prog.mo fr-FR-staff-prog.po gzip -k fr-FR-staff-prog.po du -h fr-FR-staff-prog* 824K fr-FR-staff-prog.mo 2,1M fr-FR-staff-prog.po 392K fr-FR-staff-prog.po.gz So, .mo files are smaller than .po files, but not as small as gzipped .po Which is kind of disappointing :/ Anyway, I just wanted to know if you had considered this approach. Thanks for answering.
Nice work everyone! Pushed to master for 20.05
enhancement not backported to 19.11.x branch
I tried to describe this one, but I am not compeletely sure how it fits in with the other new translation features.
Interesting idea!
I've opened Bug 27236 for the idea of packaging compressed PO files separately in a koha-i18n package...
English trivia: I was thinking that uncompress wasn't a word and that it should be decompress. However, it appears that uncompress is a word! Decompress dates back to 1905 and uncompress dates back to 1987. It even looks like uncompress actually came into usage because of computer compression. (My spellchecker doesn't like uncompress, but it appears to be wrong.) https://www.merriam-webster.com/dictionary/decompress https://www.merriam-webster.com/dictionary/uncompress