Bug 26547 - Move context from msgid to msgctxt in pref PO files
Summary: Move context from msgid to msgctxt in pref PO files
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement
Assignee: Julian Maurice
QA Contact: Testopia
URL:
Keywords:
Depends on: 25067
Blocks:
  Show dependency treegraph
 
Reported: 2020-09-26 08:30 UTC by Julian Maurice
Modified: 2025-12-18 10:48 UTC (History)
6 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 26547: Move context from msgid to msgctxt in pref PO files (9.00 KB, patch)
2020-09-26 08:31 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Move context from msgid to msgctxt in pref PO files (9.07 KB, patch)
2020-10-13 13:01 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 26547: Move context from msgid to msgctxt in pref PO files (9.07 KB, patch)
2020-10-15 10:14 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Put msgctxt into a comment as well (982 bytes, patch)
2020-10-15 10:14 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name (1.59 KB, patch)
2020-10-15 10:14 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: *DONT PUSH* Script for updating existing pref PO files (4.27 KB, patch)
2020-10-15 10:14 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Move context from msgid to msgctxt in pref PO files (9.61 KB, patch)
2025-12-16 13:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Put msgctxt into a comment as well (986 bytes, patch)
2025-12-16 13:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name (1.62 KB, patch)
2025-12-16 13:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: *DONT PUSH* Script for updating existing pref PO files (4.27 KB, patch)
2025-12-16 13:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Move context from msgid to msgctxt in pref PO files (9.61 KB, patch)
2025-12-16 15:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Put msgctxt into a comment as well (986 bytes, patch)
2025-12-16 15:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name (1.62 KB, patch)
2025-12-16 15:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: *DONT PUSH* Script for updating existing pref PO files (4.27 KB, patch)
2025-12-16 15:33 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 26547: Fix compil error (1.29 KB, patch)
2025-12-18 10:38 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2020-09-26 08:30:46 UTC
String extraction for system preferences puts context inside msgid. For instance:

    # Accounting > Policy
    msgid "accounting.pref#FinePaymentAutoPopup# automatically display "
    "a print dialog for a payment receipt when making a payment.."
    
Context should be put into msgctxt, and the filename should be in a comment, like this:
    
    #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref
    msgctxt "Accounting > Policy > FinePaymentAutoPopup"
    msgid "automatically display a print dialog for a payment receipt "
    "when making a payment.."
Comment 1 Julian Maurice 2020-09-26 08:31:24 UTC Comment hidden (obsolete)
Comment 2 Julian Maurice 2020-09-26 08:33:00 UTC
This was initially part of bug 25067, but it is better to have this in a separate bug.
Comment 3 Martin Renvoize (ashimema) 2020-10-13 13:01:24 UTC Comment hidden (obsolete)
Comment 4 Martin Renvoize (ashimema) 2020-10-13 13:02:23 UTC
Works as expected and makes sense from my perspective.

Signing off... if Bernardo or someone else very familiar with translations wanted to add an SO here I'd be happy to call it a PQA.
Comment 5 Julian Maurice 2020-10-13 13:24:16 UTC
Before pushing it, it would be nice to find a way to "convert" existing pref PO files without losing any translations. All msgids change. msgmerge try its best to not lose existing translations, but it fails on short messages like "Do"
Comment 6 Katrin Fischer 2020-10-14 21:17:40 UTC
I feel this needs more work. I tested together with bug 25067 and for de-DE ALL translations are marked fuzzy now. This would require translators in Pootle to update every and each of them. We need to see how we can avoid this.

Poedit shows: Translated 0 of 2582 (0%) - Remaining 2582

But the weird thing is: The fuzzy strings are still used in the translations so my system preference editor shows up in German.

Often times fuzzy strings can break the translation process/templates, so they should be ignored/fallback to English.

I am in a bit of trouble here - I think we need both, bug 25067 and this one to move forward. I'll put this in FQA and put my SO stamps on the other with a note.

We also need to take care of the unpackaged dependency on the other bug.
Comment 7 Julian Maurice 2020-10-15 10:14:27 UTC
Created attachment 111697 [details] [review]
Bug 26547: Move context from msgid to msgctxt in pref PO files

String extraction for system preferences was putting context inside
msgid. For instance:

    # Accounting > Policy
    msgid "accounting.pref#FinePaymentAutoPopup# automatically display "
    "a print dialog for a payment receipt when making a payment.."

Now context is put into msgctxt, and the reference is set, which is
cleaner

    #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref
    msgctxt "Accounting > Policy > FinePaymentAutoPopup"
    msgid "automatically display a print dialog for a payment receipt "
    "when making a payment.."

The downside is that some messages will have to be re-translated,
especially short messages like 'Do', for which msgmerge has a hard
time finding the corresponding new msgid.

Test plan:
1. Run `gulp po:update`
2. Verify the contents of updated pref PO files
3. Run `cd misc/translator && ./translate install <lang>`
4. Verify that sysprefs are correctly translated
5. Run `prove t/misc/translator/xgettext-pref.t`

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 8 Julian Maurice 2020-10-15 10:14:33 UTC
Created attachment 111698 [details] [review]
Bug 26547: Put msgctxt into a comment as well

Pootle does not allow to search a message by context, but it allows to
search by comment. Having the context (Tab > Section > Syspref) inside a
comment make it easier to translate all messages for a specific syspref.
Comment 9 Julian Maurice 2020-10-15 10:14:38 UTC
Created attachment 111699 [details] [review]
Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name

Example in enhanced_content.pref:

Enhanced content:
    Adlibris:
        -
            - <strong>NOTE:</strong> Using resources such as external
            images might leak sensitive data to third parties.
Comment 10 Julian Maurice 2020-10-15 10:14:46 UTC
Created attachment 111700 [details] [review]
Bug 26547: *DONT PUSH* Script for updating existing pref PO files

The script will convert pref PO files to the new format, with proper
msgctxt, reference set, and with a clean msgid
It will be needed only once before running `gulp po:update` for the
first time after applying patches of bug 26547, so this patch should not
be pushed.
This step is needed to not lose any translation.

Test plan:
1. git checkout master
2. cd misc/translator && ./translate update fr-FR
3. cp po/fr-FR-pref.po /tmp/fr-FR-pref-master.po
4. cp /tmp/fr-FR-pref-master.po /tmp/fr-FR-pref-master-updated.po
5. cd ../.. && misc/devel/update-pref-po-file.pl /tmp/fr-FR-pref-master-updated.po
6. msgcat -s /tmp/fr-FR-pref-master-updated.po > /tmp/fr-FR-pref-master-updated-sorted.po
7. cp /tmp/fr-FR-pref-master-updated.po misc/translator/po/fr-FR-pref.po
8. gulp po:update --lang fr-FR
9. msgcomm -u /tmp/fr-FR-pref-master-updated.po misc/translator/po/fr-FR-pref.po
   This should print nothing, meaning the result of `gulp:update` is the
   same as the result of update-pref-po-file.pl
10. diff -U3 /tmp/fr-FR-pref-master-updated-sorted.po misc/translator/po/fr-FR-pref.po
    The diff should be really small (empty messages removed, differences
    in line wrapping, but that's all)
11. msgattrib --only-fuzzy /tmp/fr-FR-pref-master.po
    msgattrib --only-fuzzy /tmp/fr-FR-pref-master-updated.po
    The output of those two commands should be the same (can be empty if
    there are no fuzzy messages)
12. cd misc/translator && ./translate install fr-FR
    Check that preferences are correctly translated in the staff
    interface

For the Release Manager and/or Translation Manager:

Once this patchset is pushed, all pref PO files should be updated. This
can be done immediately, with updated PO files committed directly in
master, or just before sending PO files to Pootle, and they will be
committed only before release. It is up to RM/TM.
Whatever they choose, the process is the same:
1. Apply this patch
2. Run `misc/devel/update-pref-po-file.pl misc/translator/po/*-pref.po`
3. Run `gulp po:update`
4. Discard this patch
5. Commit or send to pootle
Comment 11 Julian Maurice 2020-10-15 10:16:44 UTC
Back to needs signoff.

If you want to test, be sure to run misc/devel/update-pref-po-file.pl before any attempt to use gulp po:update
Comment 12 David Nind 2023-10-22 19:40:06 UTC
As expected, since this has been in the sign off queue for a while, the patch doesn't apply 8-(.. 

As there have been a lot of changes to the translation process over the last few releases, I was wondering how this fits in.


kohadev-koha@kohadevbox:koha(bz26547)$ git bz apply 26547

Bug 26547 - Move context from msgid to msgctxt in pref PO files

111697 - Bug 26547: Move context from msgid to msgctxt in pref PO files
111698 - Bug 26547: Put msgctxt into a comment as well
111699 - Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name
111700 - Bug 26547: *DONT PUSH* Script for updating existing pref PO files

Apply? [(y)es, (n)o, (i)nteractive] y
Applying: Bug 26547: Move context from msgid to msgctxt in pref PO files
Using index info to reconstruct a base tree...
M	misc/translator/LangInstaller.pm
M	misc/translator/xgettext-pref
A	t/misc/translator/xgettext-pref.t
Falling back to patching base and 3-way merge...
Auto-merging t/db_dependent/misc/translator/xgettext-pref.t
Auto-merging misc/translator/xgettext-pref
Auto-merging misc/translator/LangInstaller.pm
CONFLICT (content): Merge conflict in misc/translator/LangInstaller.pm
error: Failed to merge in the changes.
Patch failed at 0001 Bug 26547: Move context from msgid to msgctxt in pref PO files
Comment 13 Katrin Fischer 2023-10-29 10:46:42 UTC
Adding Jonathan because of his work on Weblate/translations.

Only a note here: we need to take care that the Do/Don't etc. are not ending up 'collapsed'. In order to form proper sentences for the system preferences they are not always translated the same way.
Comment 14 Julian Maurice 2025-12-16 13:32:14 UTC
Created attachment 190539 [details] [review]
Bug 26547: Move context from msgid to msgctxt in pref PO files

String extraction for system preferences was putting context inside
msgid. For instance:

    # Accounting > Policy
    msgid "accounting.pref#FinePaymentAutoPopup# automatically display "
    "a print dialog for a payment receipt when making a payment.."

Now context is put into msgctxt, and the reference is set, which is
cleaner

    #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref
    msgctxt "Accounting > Policy > FinePaymentAutoPopup"
    msgid "automatically display a print dialog for a payment receipt "
    "when making a payment.."

The downside is that some messages will have to be re-translated,
especially short messages like 'Do', for which msgmerge has a hard
time finding the corresponding new msgid.

Test plan:
1. Run `gulp po:update`
2. Verify the contents of updated pref PO files
3. Run `cd misc/translator && ./translate install <lang>`
4. Verify that sysprefs are correctly translated
5. Run `prove t/misc/translator/xgettext-pref.t`

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 15 Julian Maurice 2025-12-16 13:32:16 UTC
Created attachment 190540 [details] [review]
Bug 26547: Put msgctxt into a comment as well

Pootle does not allow to search a message by context, but it allows to
search by comment. Having the context (Tab > Section > Syspref) inside a
comment make it easier to translate all messages for a specific syspref.
Comment 16 Julian Maurice 2025-12-16 13:32:17 UTC
Created attachment 190541 [details] [review]
Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name

Example in enhanced_content.pref:

Enhanced content:
    Adlibris:
        -
            - <strong>NOTE:</strong> Using resources such as external
            images might leak sensitive data to third parties.
Comment 17 Julian Maurice 2025-12-16 13:32:19 UTC
Created attachment 190542 [details] [review]
Bug 26547: *DONT PUSH* Script for updating existing pref PO files

The script will convert pref PO files to the new format, with proper
msgctxt, reference set, and with a clean msgid
It will be needed only once before running `gulp po:update` for the
first time after applying patches of bug 26547, so this patch should not
be pushed.
This step is needed to not lose any translation.

Test plan:
1. git checkout master
2. cd misc/translator && ./translate update fr-FR
3. cp po/fr-FR-pref.po /tmp/fr-FR-pref-master.po
4. cp /tmp/fr-FR-pref-master.po /tmp/fr-FR-pref-master-updated.po
5. cd ../.. && misc/devel/update-pref-po-file.pl /tmp/fr-FR-pref-master-updated.po
6. msgcat -s /tmp/fr-FR-pref-master-updated.po > /tmp/fr-FR-pref-master-updated-sorted.po
7. cp /tmp/fr-FR-pref-master-updated.po misc/translator/po/fr-FR-pref.po
8. gulp po:update --lang fr-FR
9. msgcomm -u /tmp/fr-FR-pref-master-updated.po misc/translator/po/fr-FR-pref.po
   This should print nothing, meaning the result of `gulp:update` is the
   same as the result of update-pref-po-file.pl
10. diff -U3 /tmp/fr-FR-pref-master-updated-sorted.po misc/translator/po/fr-FR-pref.po
    The diff should be really small (empty messages removed, differences
    in line wrapping, but that's all)
11. msgattrib --only-fuzzy /tmp/fr-FR-pref-master.po
    msgattrib --only-fuzzy /tmp/fr-FR-pref-master-updated.po
    The output of those two commands should be the same (can be empty if
    there are no fuzzy messages)
12. cd misc/translator && ./translate install fr-FR
    Check that preferences are correctly translated in the staff
    interface

For the Release Manager and/or Translation Manager:

Once this patchset is pushed, all pref PO files should be updated. This
can be done immediately, with updated PO files committed directly in
master, or just before sending PO files to Pootle, and they will be
committed only before release. It is up to RM/TM.
Whatever they choose, the process is the same:
1. Apply this patch
2. Run `misc/devel/update-pref-po-file.pl misc/translator/po/*-pref.po`
3. Run `gulp po:update`
4. Discard this patch
5. Commit or send to pootle
Comment 18 Jonathan Druart 2025-12-16 15:22:17 UTC
It still does not apply for me.
Comment 19 Julian Maurice 2025-12-16 15:33:25 UTC
Created attachment 190548 [details] [review]
Bug 26547: Move context from msgid to msgctxt in pref PO files

String extraction for system preferences was putting context inside
msgid. For instance:

    # Accounting > Policy
    msgid "accounting.pref#FinePaymentAutoPopup# automatically display "
    "a print dialog for a payment receipt when making a payment.."

Now context is put into msgctxt, and the reference is set, which is
cleaner

    #: koha-tmpl/intranet-tmpl/prog/en/modules/admin/preferences/accounting.pref
    msgctxt "Accounting > Policy > FinePaymentAutoPopup"
    msgid "automatically display a print dialog for a payment receipt "
    "when making a payment.."

The downside is that some messages will have to be re-translated,
especially short messages like 'Do', for which msgmerge has a hard
time finding the corresponding new msgid.

Test plan:
1. Run `gulp po:update`
2. Verify the contents of updated pref PO files
3. Run `cd misc/translator && ./translate install <lang>`
4. Verify that sysprefs are correctly translated
5. Run `prove t/misc/translator/xgettext-pref.t`

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 20 Julian Maurice 2025-12-16 15:33:27 UTC
Created attachment 190549 [details] [review]
Bug 26547: Put msgctxt into a comment as well

Pootle does not allow to search a message by context, but it allows to
search by comment. Having the context (Tab > Section > Syspref) inside a
comment make it easier to translate all messages for a specific syspref.
Comment 21 Julian Maurice 2025-12-16 15:33:28 UTC
Created attachment 190550 [details] [review]
Bug 26547: Fix msgctxt in pref PO file for blocks without a pref name

Example in enhanced_content.pref:

Enhanced content:
    Adlibris:
        -
            - <strong>NOTE:</strong> Using resources such as external
            images might leak sensitive data to third parties.
Comment 22 Julian Maurice 2025-12-16 15:33:30 UTC
Created attachment 190551 [details] [review]
Bug 26547: *DONT PUSH* Script for updating existing pref PO files

The script will convert pref PO files to the new format, with proper
msgctxt, reference set, and with a clean msgid
It will be needed only once before running `gulp po:update` for the
first time after applying patches of bug 26547, so this patch should not
be pushed.
This step is needed to not lose any translation.

Test plan:
1. git checkout master
2. cd misc/translator && ./translate update fr-FR
3. cp po/fr-FR-pref.po /tmp/fr-FR-pref-master.po
4. cp /tmp/fr-FR-pref-master.po /tmp/fr-FR-pref-master-updated.po
5. cd ../.. && misc/devel/update-pref-po-file.pl /tmp/fr-FR-pref-master-updated.po
6. msgcat -s /tmp/fr-FR-pref-master-updated.po > /tmp/fr-FR-pref-master-updated-sorted.po
7. cp /tmp/fr-FR-pref-master-updated.po misc/translator/po/fr-FR-pref.po
8. gulp po:update --lang fr-FR
9. msgcomm -u /tmp/fr-FR-pref-master-updated.po misc/translator/po/fr-FR-pref.po
   This should print nothing, meaning the result of `gulp:update` is the
   same as the result of update-pref-po-file.pl
10. diff -U3 /tmp/fr-FR-pref-master-updated-sorted.po misc/translator/po/fr-FR-pref.po
    The diff should be really small (empty messages removed, differences
    in line wrapping, but that's all)
11. msgattrib --only-fuzzy /tmp/fr-FR-pref-master.po
    msgattrib --only-fuzzy /tmp/fr-FR-pref-master-updated.po
    The output of those two commands should be the same (can be empty if
    there are no fuzzy messages)
12. cd misc/translator && ./translate install fr-FR
    Check that preferences are correctly translated in the staff
    interface

For the Release Manager and/or Translation Manager:

Once this patchset is pushed, all pref PO files should be updated. This
can be done immediately, with updated PO files committed directly in
master, or just before sending PO files to Pootle, and they will be
committed only before release. It is up to RM/TM.
Whatever they choose, the process is the same:
1. Apply this patch
2. Run `misc/devel/update-pref-po-file.pl misc/translator/po/*-pref.po`
3. Run `gulp po:update`
4. Discard this patch
5. Commit or send to pootle
Comment 23 Julian Maurice 2025-12-17 08:35:33 UTC
(In reply to Jonathan Druart from comment #18)
> It still does not apply for me.

Patches can be applied with git-bz. It's a perl-git-bz bug (probably related to encoding)
Comment 24 Jonathan Druart 2025-12-18 10:37:48 UTC
I am stuck at step 6 of the last patch

6. msgcat -s /tmp/fr-FR-pref-master-updated.po > /tmp/fr-FR-pref-master-updated-sorted.po

$ msgcat -s /tmp/fr-FR-pref-master-updated.po > /tmp/fr-FR-pref-master-updated-sorted.po
/tmp/fr-FR-pref-master-updated.po:21297: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:21305: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:21391: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:21399: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:21407: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:21706: duplicate message definition...
/tmp/fr-FR-pref-master-updated.po:1078: ...this is the location of the first definition
/tmp/fr-FR-pref-master-updated.po:21712: duplicate message definition...
/tmp/fr-FR-pref-master-updated.po:1372: ...this is the location of the first definition
/tmp/fr-FR-pref-master-updated.po:23381: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:23389: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:23667: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:23675: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:23683: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:23691: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:23699: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:24409: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:24417: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:24425: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:24439: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:24447: inconsistent use of #~
/tmp/fr-FR-pref-master-updated.po:24455: inconsistent use of #~
msgcat: too many errors, aborting
Comment 25 Jonathan Druart 2025-12-18 10:38:32 UTC
Created attachment 190613 [details] [review]
Bug 26547: Fix compil error

Patch from commit 4ee3e62
Comment 26 Jonathan Druart 2025-12-18 10:46:09 UTC
I think we don't need to adjust msgid.

I would remove the comment with a sed before the update. The only problem is removing the fuzzy. We should remove only for the new occurrences, not that ones that were fuzzy before.
Comment 27 Jonathan Druart 2025-12-18 10:48:30 UTC
Additionally there are some QA failures:

[FAIL] misc/translator/LangInstaller.pm
   FAIL   tidiness
                File is not tidy, please run `perl misc/devel/tidy.pl misc/translator/LangInstaller.pm`
[FAIL] t/db_dependent/misc/translator/xgettext-pref.t
   FAIL   tidiness
                File is not tidy, please run `perl misc/devel/tidy.pl t/db_dependent/misc/translator/xgettext-pref.t`

I am also not 100% sure about my "fix compile error", I will need to read again why we added that on bug 31640.