Created attachment 162490 [details] Russian interface After the last update of the program, the display of user pages, as well as the interface for writing off and adding literature, disappeared in the Russian administrator interface. Everything works in the English interface.
Could you provide instructions for how to reproduce the problem?
You need to switch the interface language to Russian and open the card of any user.
Template process failed: file error - parse error - member-display-address-style.inc line 1: unexpected token (line_break) [% КОМПЛЕКТ line_break = "<br/>" %] at /kohadevbox/koha/C4/Templates.pm line 134 It looks like a template directive is being picked up for translation. This is the line from the English language template: [%~ SET line_break = "<br/>" ~%] A temporary fix would be to edit the affected file, member-display-address-style.inc, to replace "КОМПЛЕКТ" with "SET" It looks like the same problem is in display-library-address.inc. The real fix will require that we figure out why the translation script is picking that line up.
(In reply to Owen Leonard from comment #3) > It looks like the same problem is in display-library-address.inc. I should have pointed out that this include is in the OPAC, not the staff interface.
Created attachment 162516 [details] [review] Bug 36171: Use FILTER collapse instead of tilde in address includes A couple of our include files used for displaying addresses have "~" added to every template directive in order to collapse whitespace in the generated HTML. This seems to cause problems for the translation script which was picking up the "SET" in "[% SET foo = 'bar' %]. This patch modifies the affected files and related files (for consistency) to use '[% FILTER collapse %]'. This seems to fix the problem. Note: This patch makes indentation changes, so ignore whitespace when viewing the diff. To test, apply the patch and search for a patron in the staff interface. - View the patron detail page. All addresses, including alternate address and alternative contact address, should be displayed correctly. - Edit the patron and confirm that the forms for entering those addresses also look correct. - Go to Administration -> System preferences and test using other options for the "AddressFormat" preference. - In the OPAC, view the "Libraries" page. Confirm that addresses are displayed correctly. - Rebuild the ru-RU templates to confirm that there is no error: > perl misc/translator/translate update ru-RU > perl misc/translator/translate install ru-RU - Enable the language, if necessary, by updating the "language" and "OPACLanguage" system preferences. - Repeat the testing steps above.
The patch works. But, as soon as I update the language according to the last point, the problem returns. This happened after the last update.
When will this error be fixed in the next update?
I'm putting this back to "NEW" because I've been unable to find a template-level fix for it.
(In reply to sergeinik from comment #7) > When will this error be fixed in the next update? This bug was trying to find a solution so that these text no longer shows up in translations. A temporary fix is that someone with translation permissions for Russian fixes these: https://translate.koha-community.org/search/koha/-/ru/?q=+source%3ASET+&sort_by=-priority%2Cposition&checksum=
I've fixed the broken translation with Jonathan's help.
Created attachment 165506 [details] [review] Bug 36171: Incorrect translation of Template Toolkit directive causes patron view error in ru-RU It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. The template toolkit tags should probably be parsed first, but as a workaround the HTML tags in a template toolkit tag can be obfuscated. This patch uses concatenation as an example. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file.
I think this is supposed to be "Needs signoff" ?
It is just a workaround, so I don't know if it is ready to be signed off. Is it enough or should the method for parsing be discussed?
This feels hacky. IMO we should fix it properly (ie. not pick those strings for translation).
A couple of questions: 1. Does this only affect ru-RU, or all languages? 2. Is the status still "Critical"?
> 1. Does this only affect ru-RU, or all languages? It should affect all languages where the problematic strings have been translated. Here is what I've been able to find https://translate.koha-community.org/translate/koha/23-11/opac-bootstrap/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3ASET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-11/opac-bootstrap/te/?checksum=28bf1f208fcb60c4&q=+source%3ASET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/main/opac-bootstrap/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3ASET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/main/opac-bootstrap/te/?checksum=28bf1f208fcb60c4&q=+source%3ASET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/main/staff-prog/te/?checksum=28bf1f208fcb60c4&q=+source%3ASET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/24-05/opac-bootstrap/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/opac-bootstrap/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/opac-bootstrap/te/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/opac-bootstrap/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/staff-prog/te/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/te/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-11/staff-prog/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-11/staff-prog/te/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/24-05/staff-prog/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/main/staff-prog/fa_ARAB/?checksum=28bf1f208fcb60c4&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/staff-prog/ca/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/staff-prog/fa_ARAB/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/staff-prog/te/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/ca/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/fa_ARAB/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/te/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition > 2. Is the status still "Critical"? It seems fair until someone with rights on these languages/all languages empties the translation or copy paste the source. Maybe the 2nd one is best? That's the 1st group of strings. 28bf1f208fcb60c4 The 2nd group (47311df64b1b946c) needs only to have the variable name fixed. And then for digging at the root cause could be major severity I guess. ---- (In reply to Jonathan Druart from comment #14) > This feels hacky. IMO we should fix it properly (ie. not pick those strings > for translation). +1 to not pick or eliminate the uses of these weird conditions to insert line breaks and generate links 1) If not picking those strings, it must be kept in mind that the second group of strings with breaking translations (47311df64b1b946c) is needed to be translated: [%% SET countries = [ "Afghanistan", "Albania", Thankfully it's not needed anymore since 23.11 thanks to Bug 35091 :D So if someone manages to exclude all "SET" TT tags it could be applied to main, 24.05, 23.11 and either we do not bother with older releases (just fix the existing error in the links) or apply the current patch there. 2) If eliminating the structure are get picked up: about display-library-address.inc and member-display-address-style.inc Is that even needed? It might be dead code. There are two [% PROCESS 'display-library-address' %] And I don't see how the no_line_break variable would be set. About pref_MARCOverlayRules_link it's from bug 29589 and likely pref_CurbsidePickup_link is modeled after it. bug 29589 was already to fix a translatability issue ^^" Since then, is there know another trick known that could superseed the solution of bug 29589 ? But maybe we are not seeing any translation error on pref_MARCOverlayRules_link and pref_CurbsidePickup_link because the final are scary enough that no one thinks it's really something to translate ^^ %s %s [%% SET pref_MARCOverlayRules_link = ' %s [%% SET pref_CurbsidePickup_link = ' So maybe we can get away with these link generation cases?
(In reply to Victor Grousset/tuxayo from comment #16) > > 2. Is the status still "Critical"? > > It seems fair until someone with rights on these languages/all languages > empties the translation or copy paste the source. Maybe the 2nd one is best? > That's the 1st group of strings. 28bf1f208fcb60c4 > > The 2nd group (47311df64b1b946c) needs only to have the variable name fixed. So I made suggestions to string found broken, they just need to be accepted by someone with rights on all languages or just rights on either Telugu, Persian and Catalan. ----- > But maybe we are not seeing any translation error on > pref_MARCOverlayRules_link and pref_CurbsidePickup_link because the final > are scary enough that no one thinks it's really something to translate ^^ *the strings are scary enough
I've accepted all your suggestions.
Created attachment 170194 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser
Created attachment 170195 [details] Template toolkit tags repaired
Here is an attempt at repairing split template toolkit tags. If a text block begins with the end of a tag, it goes back through the tokens to try to find the beginning and combine them into a directive token. I also included a list of the "repairs". It looks like the action block for the virtual shelves results would need to use I18N.t for translation and doc-head-open would need to use Dumper.dump_html instead of filtering (the '-->' is confusing the parser). If this is not reasonable, I can limit it to SET tags. Another option is to slurp the file, remove the tags, and then use $self->parse instead of $self->parse_file. I would appreciate any feedback and testing since this was done on a Friday afternoon.
Created attachment 170227 [details] Bug 36171: Fix template toolkit tags split by HTML parser Bug 36171: Fix template toolkit tags split by HTML parser It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. This patch searches for text sections that have a template toolkit end marker without a start and then searches for the start in previous sections. It then creates a new template toolkit section with the start and end markers. It also changes the replacement of "-->" in doc-head-open.inc so it is not interpreted as the end of the comment. It now replaces all "--" with "- -". The action buttons in the staff Lists view have also been changed to allow translation. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file. You can also add translations for "Edit", "Delete", and "Transfer" in messages.po to make sure that the action buttons are translated in the Lists view.
Created attachment 170262 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. This patch searches for text sections that have a template toolkit end marker without a start and then searches for the start in previous sections. It then creates a new template toolkit section with the start and end markers. It also changes the replacement of "-->" in doc-head-open.inc so it is not interpreted as the end of the comment. It now replaces all "--" with "- -". The action buttons in the staff Lists view have also been changed to allow translation. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file. You can also add translations for "Edit", "Delete", and "Transfer" in messages.po to make sure that the action buttons are translated in the Lists view.
(In reply to Katrin Fischer from comment #18) > I've accepted all your suggestions. Thanks a lot :) Maybe it's the issues with the translation platform that happened just after that, but these are still pending: https://translate.koha-community.org/translate/koha/22-11/staff-prog/ca/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/staff-prog/fa_ARAB/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/22-11/staff-prog/te/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/ca/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/fa_ARAB/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition https://translate.koha-community.org/translate/koha/23-05/staff-prog/te/?checksum=47311df64b1b946c&q=+source%3A%25+SET+&sort_by=-priority%2Cposition
Thanks for the links, I had only been looking for the line_break one. Should be good now.
(In reply to Kevin Carnes from comment #23) > It also changes the replacement of "-->" in doc-head-open.inc so it is not > interpreted as the end of the comment. It now replaces all "--" with "- -". Why all?
Created attachment 170342 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. This patch searches for text sections that have a template toolkit end marker without a start and then searches for the start in previous sections. It then creates a new template toolkit section with the start and end markers. The action buttons in the staff Lists view have been changed to allow translation. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file. You can also add translations for "Edit", "Delete", and "Transfer" in messages.po to make sure that the action buttons are translated in the Lists view.
(In reply to Jonathan Druart from comment #26) > (In reply to Kevin Carnes from comment #23) > > It also changes the replacement of "-->" in doc-head-open.inc so it is not > > interpreted as the end of the comment. It now replaces all "--" with "- -". > > Why all? Just in case there was a browser that was strict about an even number of "--" and also to avoid writing '-' _ '->'. I removed the change since it looks like everything between a comment and an HTML tag is ignored anyways, so the change is not necessary. In this case it doesn't cause a problem, but there is a problem if a template toolkit tag is outside a comment and contains a "<!". There is also a problem if it is inside a comment and contains a "-->" followed by an HTML tag and then some text before the real end of the comment. Finally, any text between an HTML tag and a comment is ignored (e.g. <a>before<!-- comment -->after</a>).
Whats the current state of this?
Since the broken translations have been fixed on weblate, it's not critical anymore. Now back to fixing the root issue to avoid the possibility to make broken translations. Early QA feedback seems to have been addressed, so it looks ready for testing.
Created attachment 172327 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. This patch searches for text sections that have a template toolkit end marker without a start and then searches for the start in previous sections. It then creates a new template toolkit section with the start and end markers. The action buttons in the staff Lists view have been changed to allow translation. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file. You can also add translations for "Edit", "Delete", and "Transfer" in messages.po to make sure that the action buttons are translated in the Lists view. Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment on attachment 170342 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser Errata: I attached the version that I applied to 23.05 which was the branch with more problematic cases than main. (version for main coming)
Created attachment 172334 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. This patch searches for text sections that have a template toolkit end marker without a start and then searches for the start in previous sections. It then creates a new template toolkit section with the start and end markers. The action buttons in the staff Lists view have been changed to allow translation. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file. You can also add translations for "Edit", "Delete", and "Transfer" in messages.po to make sure that the action buttons are translated in the Lists view. Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Tested on 23.05 to get the two observed cases of extracted strings that can break the UI (I also applied the test plan there): #~ msgid "%s %s %s [%%~ SET line_break = \"" It's successfully not picked anymore. That solves 70% of the found cases of translations breaking pages. It's still picking up this: #: koha-tmpl/intranet-tmpl/prog/en/includes/country-list.inc:1 #, c-format msgid "[%% SET countries = [ \"Afghanistan\", \"Albania\", \"Algeria\", [...] Which allows translation of country name in Koha <= 23.05 (before bug 35091) But it's picked in it's entirety. So with "countries" which was the cause of breakage when it's translated.(30% of the observed breakages) If it would somehow be done differently, then the string would change to not have "countries". But then the whole list would have to be retranslated for old releases which might not have much translation attention. Which is the same downside with a less good result than backporting bug 35091. So not much do to for these cases. Not and issue for this ticket. --- More testing: Looking at the extracted string changes, there are more strings that are not extracted anymore. -msgid "%s %s %s %s %s %s %s %s %s %s %s%s %s %s %s %s [%% # [ WRAPPER tab_item tabname= \"tab_name_1\" bt_active= 1 ] " -msgid "[ END ] %%] %s %s %s %s %s %s %s %s %s %s " -msgid "%s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s %s%s%s %s %s %s %s %s [%%- span_start = '" -msgid "[%% HOLIDAYS_LOO.DESCRIPTION.replace('\\\\\\r\\\\\\n', '" [... other stuff] Nice, good riddance :) --- [% # [ WRAPPER tab_item tabname= "tab_name_1" bt_active= 1 ] <span>Tab text 1</span> [ END ] %] [% # [ WRAPPER tab_item tabname= "tab_name_2" ] <span>Tab text 2</span> [ END ] %] [% # <a href="/cgi-bin/koha/module/module-home.pl"> Module </a> %] [% # <a href="/cgi-bin/koha/module/page.pl"> Module page </a> %] [% # <span>Module page</span> %] [% # <span>Add form</span> %] Very good! This stuff that was in comments isn't picked up anymore! :D --- -msgid "%s [%% SET pref_CurbsidePickup_link = '" -msgid "%s %s [%% SET pref_MARCOverlayRules_link = '" Nice, it takes care of these cases that were identified as maybe being a pain! :)
It works! :)
QA note: really don't miss Comment 34 to avoid duplicating some tedious testing and po diff inspection and findings on issues that are buried much earlier in comments. And time of running `misc/translator/translate update` doesn't change in a meaningful way (it's within variance)
Created attachment 173252 [details] [review] Bug 36171: Fix template toolkit tags split by HTML parser It appears that the template parser parses HTML tags first and then template toolkit tags. This can cause a template toolkit tag that contains an HTML tag to have its start and end markers separated. This causes it to be treated as text that can be translated instead of a template toolkit tag. This patch searches for text sections that have a template toolkit end marker without a start and then searches for the start in previous sections. It then creates a new template toolkit section with the start and end markers. The action buttons in the staff Lists view have been changed to allow translation. To test: 1. You will need to edit the staff-prog.po for a language and modify the the translation for "SET line_break". You can change SET to FOO. 2. Install the translation for the language. 3. View the details of a patron in the staff interface with the language. 4. The page will have an error. 5. Apply the patch. 6. Install the translation for the language. 7. View the details of a patron in the staff interface with the language. 8. The page should now display correctly. If you want to, you can run "gulp po:update --lang LANGUAGE-CODE" to make sure that the template tag is no longer included in .po file. You can also add translations for "Edit", "Delete", and "Transfer" in messages.po to make sure that the action buttons are translated in the Lists view. Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Amended-by: Jonathan Druart Tidy Signed-off-by: Jonathan Druart <jonathan.druart@bugs.koha-community.org>
Nice one, Kevin!
Could we have a unit test here to highlight the change please?
(In reply to Katrin Fischer from comment #39) > Could we have a unit test here to highlight the change please? I haven't requested it because nothing of our translation script is tested. I don't think it's fair to request one here.
This is not done yet, please don't adjust the status.
(In reply to Jonathan Druart from comment #41) > This is not done yet, please don't adjust the status. I was going to push it, but I thought I'd still ask :) I have it on my branch and it will go in with the next batch.
Pushed for 24.11! Well done everyone, thank you!
Backported to 24.05.x for upcoming 24.05.06
While this was in 24.11, I've added a release note as it will be back ported to 24.05.x, which will have release notes generated. I hope I've got the description right, please update if I haven't.
Pushed to 23.11.x for 23.11.11
Not pushed to LTS. Marked Resolved. If you feel this should be in LTS please reply with your reason.