Bug 34983

Summary: Retranslating causes changes in locale_data.json
Product: Koha Reporter: Marcel de Rooy <m.de.rooy>
Component: Architecture, internals, and plumbingAssignee: Marcel de Rooy <m.de.rooy>
Status: CLOSED FIXED QA Contact: Kyle M Hall (khall) <kyle>
Severity: enhancement    
Priority: P5 - low CC: fridolin.somers, kyle
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=34959
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.11.00,23.05.05,22.11.12
Circulation function:
Attachments: Bug 34983: Force a sort order in JSON output of po2json
Bug 34983: Force a sort order in JSON output of po2json
Bug 34983: Force a sort order in JSON output of po2json

Description Marcel de Rooy 2023-10-04 13:03:55 UTC
Actually, nothing may have changed. But the hash keys are processed in random order. We could force ordering them at a slightly slower rate. Advantage would be to see only changes when translations really changed.

Actually, the performance changes that I saw were neglectible. The staff po file is converted in about 600-700 ms both ways. Sometimes the one is faster than the other even.
Comment 1 Marcel de Rooy 2023-10-04 13:18:36 UTC
Created attachment 156538 [details] [review]
Bug 34983: Force a sort order in JSON output of po2json

This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.

Test plan:
Copy a staff PO file from misc/translator to test.po
Now run:   time misc/translator/po2json test.po > json1
And do:    time misc/translator/po2json test.po > json2
Run:       diff json1 json2; #They should be the same.
Now:       export CANONICAL=0
And run:   time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run:   diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 2 Marcel de Rooy 2023-10-04 13:20:15 UTC
if you test, please check the times reported for the various runs. If you see something unusual, please copy it on the report here. Thanks.
Comment 3 David Nind 2023-10-08 19:13:54 UTC
Created attachment 156712 [details] [review]
Bug 34983: Force a sort order in JSON output of po2json

This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.

Test plan:
Copy a staff PO file from misc/translator to test.po
Now run:   time misc/translator/po2json test.po > json1
And do:    time misc/translator/po2json test.po > json2
Run:       diff json1 json2; #They should be the same.
Now:       export CANONICAL=0
And run:   time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run:   diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>
Comment 4 Kyle M Hall (khall) 2023-10-13 13:18:58 UTC
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json1

real	0m0.022s
user	0m0.011s
sys	0m0.011s
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json2

real	0m0.022s
user	0m0.015s
sys	0m0.008s
kyle@docker:~/repos/koha$ diff json1 json2
kyle@docker:~/repos/koha$ export CANONICAL=0
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json1

real	0m0.022s
user	0m0.013s
sys	0m0.009s
kyle@docker:~/repos/koha$ time misc/translator/po2json /home/kyle/repos/koha/misc/translator/po/zh-Hant-TW-messages.po > json2

real	0m0.023s
user	0m0.015s
sys	0m0.008s
Comment 5 Kyle M Hall (khall) 2023-10-13 13:19:53 UTC
Created attachment 157084 [details] [review]
Bug 34983: Force a sort order in JSON output of po2json

This patch does so by default. If you do not want them sorted,
export CANONICAL-0 on the command line.

Test plan:
Copy a staff PO file from misc/translator to test.po
Now run:   time misc/translator/po2json test.po > json1
And do:    time misc/translator/po2json test.po > json2
Run:       diff json1 json2; #They should be the same.
Now:       export CANONICAL=0
And run:   time misc/translator/po2json test.po > json3
And again: time misc/translator/po2json test.po > json4
And run:   diff json3 json4; # Lots of changes
Remove the created cruft. And signoff :)

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 6 Tomás Cohen Arazi (tcohen) 2023-10-24 13:06:49 UTC
Pushed to master for 23.11.

Nice work everyone, thanks!
Comment 7 Fridolin Somers 2023-10-26 06:54:48 UTC
I bet we need to backport for new translation system, like Bug 34959

Pushed to 23.05.x for 23.05.05
Comment 8 Matt Blenkinsop 2023-11-13 15:11:25 UTC
Nice work everyone!

Pushed to oldstable for 22.11.x