Bug 34983 - Retranslating causes changes in locale_data.json
Summary: Retranslating causes changes in locale_data.json
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Architecture, internals, and plumbing (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Marcel de Rooy
QA Contact: Kyle M Hall
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-10-04 13:03 UTC by Marcel de Rooy
Modified: 2023-11-13 15:11 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.11.00,23.05.05,22.11.12


Attachments
Bug 34983: Force a sort order in JSON output of po2json (1.39 KB, patch)
2023-10-04 13:18 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 34983: Force a sort order in JSON output of po2json (1.39 KB, patch)
2023-10-08 19:13 UTC, David Nind
Details | Diff | Splinter Review
Bug 34983: Force a sort order in JSON output of po2json (1.45 KB, patch)
2023-10-13 13:19 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Marcel de Rooy 2023-10-04 13:03:55 UTC
Actually, nothing may have changed. But the hash keys are processed in random order. We could force ordering them at a slightly slower rate. Advantage would be to see only changes when translations really changed.

Actually, the performance changes that I saw were neglectible. The staff po file is converted in about 600-700 ms both ways. Sometimes the one is faster than the other even.
Comment 1 Marcel de Rooy 2023-10-04 13:18:36 UTC
Created attachment 156538 [details] [review]
Bug 34983: Force a sort order in JSON output of po2json

This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.

Test plan:
Copy a staff PO file from misc/translator to test.po
Now run:   time misc/translator/po2json test.po > json1
And do:    time misc/translator/po2json test.po > json2
Run:       diff json1 json2; #They should be the same.
Now:       export CANONICAL=0
And run:   time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run:   diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 2 Marcel de Rooy 2023-10-04 13:20:15 UTC
if you test, please check the times reported for the various runs. If you see something unusual, please copy it on the report here. Thanks.
Comment 3 David Nind 2023-10-08 19:13:54 UTC
Created attachment 156712 [details] [review]
Bug 34983: Force a sort order in JSON output of po2json

This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.

Test plan:
Copy a staff PO file from misc/translator to test.po
Now run:   time misc/translator/po2json test.po > json1
And do:    time misc/translator/po2json test.po > json2
Run:       diff json1 json2; #They should be the same.
Now:       export CANONICAL=0
And run:   time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run:   diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Comment 4 Kyle M Hall 2023-10-13 13:18:58 UTC
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json1

real	0m0.022s
user	0m0.011s
sys	0m0.011s
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json2

real	0m0.022s
user	0m0.015s
sys	0m0.008s
kyle@docker:~/repos/koha$ diff json1 json2
kyle@docker:~/repos/koha$ export CANONICAL=0
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json1

real	0m0.022s
user	0m0.013s
sys	0m0.009s
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json2

real	0m0.023s
user	0m0.015s
sys	0m0.008s
Comment 5 Kyle M Hall 2023-10-13 13:19:53 UTC
Created attachment 157084 [details] [review]
Bug 34983: Force a sort order in JSON output of po2json

This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.

Test plan:
Copy a staff PO file from misc/translator to test.po
Now run:   time misc/translator/po2json test.po > json1
And do:    time misc/translator/po2json test.po > json2
Run:       diff json1 json2; #They should be the same.
Now:       export CANONICAL=0
And run:   time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run:   diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 6 Tomás Cohen Arazi 2023-10-24 13:06:49 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 7 Fridolin Somers 2023-10-26 06:54:48 UTC
I bet we need to backport for new translation system, like Bug 34959

Pushed to 23.05.x for 23.05.05
Comment 8 Matt Blenkinsop 2023-11-13 15:11:25 UTC
Nice work everyone!

Pushed to oldstable for 22.11.x