Bug 37472

Created attachment 169561 [details] [review]
Bug 37472: Enable extraction of strings from plugins

This patch extends the existing gulpfile to enable extraction of strings from plugins using a --plugins flag. Strings are extracted to files inside the plugin directory to then be translated via pull requests

Created attachment 169562 [details] [review]
Bug 37472: Allow plugin po files to be installed

This patch allows the koha-translate command to identify plugins with translation files and install them alongside the core translation files

Created attachment 169563 [details] [review]
Bug 37472: Trigger translations from plugin install

This patch allows a plugin installation to update translated languages to include any translation files included in the plugin

Created attachment 169564 [details] [review]
Bug 37472: Tidy LangInstaller file

The idea of the work here is to take the existing workflows that we use to translate core Koha and extend them to be able to work for plugins. There are two parts to this:

1. We need to extract strings from plugins in development mode and create po files that can then be translated through pull requests and re-released. (Long term the plan will be to integrate this into the work being done to create a centralised plugin store. Plugins admitted to the store could possibly be integrated into weblate and translated through there).
2. We need to be able to extract newly translated po files from a plugin and install them alongside any languages already installed on the system.

I’ve tried to make sure there are as few changes as possible required in the plugins themselves to make rebasing existing plugins easier. Effectively the only thing that will need doing is ensuring that any template files are stored in a ‘views/en/‘ folder to mirror the template directory structure we use in core. This is because the templates will be fetched by get_template_and_user which substitutes the ‘en’ part of the file path with the desired language.

To test this I’ve forked the kitchen sink plugin as this could then be merged if this solution is workable and gets pushed to main. We would then have translation examples in the kitchen sink plugin already. The only changes made to the plugin itself are moving the template files as described above and also adding a new javascript file to demonstrate javascript translations.

To test:

1. Clone the fork from: https://github.com/mblenk/dev-koha-plugin-kitchen-sink into your plugins folder that you would use for ktd --plugins up and checkout the plugin_translations branch
2. Apply the patches from the bug and run ktd --plugins up to load up ktd with the plugin installed .
3. Navigate to the plugins page in administration to check the plugin has installed correctly during ktd start up. You may need to run restart_all if it is showing an error.
4. Open up the directory for the cloned kitchen sink fork and note the new configure.js file that has been added, then have a look at configure.tt . I have added some new strings at the bottom in a list with the id of “js_list”. One of these is a template string, one is injected from a script tag just beneath and the third comes from configure.js
5. To extract strings from core koha we use gulp commands like gulp po:create. I have added a new flag to this to handle plugin strings - run the following command to extract the strings: gulp po:create  --lang de-DE --plugins
6. Look in the plugin directory and observe that a new folder has been created called translator. Inside this is a folder called ‘po’ and inside you will find the template .pot files as well as a folder for de-DE with two .po files in there. One is for javascript, one for template strings. Open these up and observe that the strings have been extracted from files and are available to translate
7. Make some translations for the items in the list on configure.tt that we looked at in step 4 as well as any other translations that take your fancy.
8. Now we’re going to install the language to see if the plugin translates: koha-translate --install de-DE --dev kohadev
9. In the plugin directory under the views folder, you will see that a new folder for de-DE has been created with translated .po files complete with the translations from step 7.
10. In system preferences, enable the use of German as a language in the interface
11. Go the plugins page and on the dropdown menu for the kitchen sink plugin click on ‘Konfigurieren’ to go the configure page
12. Your translations should be visible on the page when German is the selected language
13. We’ve proven the concept when installing a language with a plugin already installed, we now need to prove the reverse i.e. that installing a plugin on a system with languages already installed will update any existing language translations
14. Run: koha-translate --remove de-DE --dev kohadev
15. In the dev tools, find the cookie for Opaclanguages and delete it
16. Run reset_all
17. As far as Koha is concerned the both the plugin and the language are now no longer installed so we can now reinstall the language.
18. Repeat step 8
19. Repeat step 10
20. Run perl misc/devel/install_plugins.pl
21. You should see some of the logging from the language installation we saw earlier (there is also a broken pipe error that needs tracking down and silencing but this is harmless).
22. Repeat step 11
23. The translations should work as before meaning we now have a translatable plugin!

Hi Matt, 

first of all: thanks for taking this on!

I have tried to understand how this will work from the test plan, but I am not sure I got it completely right yet.

I assume that the idea is to not copy the template files to a de-DE directory, but handle all the translations on the fly. Is this correct?

In that case I am not sure why an install translation step for plugins would be needed, as I assume they would come with all available languages po files as part of their source. What is achieved by installing a language for a plugin?

Ideally I think if a language has been installed for staff or OPAC these translations should be made automatically available when the plugin is installed without requiring additional steps from server side. 

I am also not sure what is meant here:

13. We’ve proven the concept when installing a language with a plugin already installed, we now need to prove the reverse i.e. that installing a plugin on a system with languages already installed will update any existing language translations

I suspect I am missing something big here, please bear with me :)

Thanks for taking a look!
 
> I assume that the idea is to not copy the template files to a de-DE
> directory, but handle all the translations on the fly. Is this correct?

The translated template files are copied into a de-DE directory in the plugin directory rather than being added to all the existing core translated files. This is so that if a plugin is ever uninstalled it removes the translated files and they're not hanging around with the core templates. We use the de-DE directory that now sits in the plugin to substitute the translated templates, the same way as we do in core Koha


> In that case I am not sure why an install translation step for plugins would
> be needed, as I assume they would come with all available languages po files
> as part of their source. What is achieved by installing a language for a
> plugin?

The plugins contain the po files but not the translated templates. The install step is needed to translate these using the core translation code, rather than needing to write translation code into the plugins themselves. It also takes any javascript strings and adds them to the existing javascript translations. (The plugin actually handles the translation step itself, see below)
 
> Ideally I think if a language has been installed for staff or OPAC these
> translations should be made automatically available when the plugin is
> installed without requiring additional steps from server side. 

When a plugin is installed it won't require server side steps, it will trigger the translations during install (see below for the two plugin translation scenarios)
 
> I am also not sure what is meant here:
> 
> 13. We’ve proven the concept when installing a language with a plugin
> already installed, we now need to prove the reverse i.e. that installing a
> plugin on a system with languages already installed will update any existing
> language translations
> 
> I suspect I am missing something big here, please bear with me :)

There are two key scenarios that could happen:
1) A plugin is already installed but no language has ever been installed on the system. When the language is installed, it needs to also account for any existing plugins that have appropriate translation files. This is the first half of the test plan up to step 13

2) A system already has additional languages installed and a new plugin is installed on the system that has translation files for those languages. The plugin installation needs to ensure that it is also translating itself to match the languages installed on the system. This is covered in the second half of the test plan when we uninstall the plugin and try re-installing it after we have installed a language on the system. In this scenario we don't need to run koha-translate on the system like you queried in your second question above as the plugin installation handles it for us

Hope this all makes sense, let me know if you have any more questions!

(In reply to Matt Blenkinsop from comment #7)
> Thanks for taking a look!
>  
> > I assume that the idea is to not copy the template files to a de-DE
> > directory, but handle all the translations on the fly. Is this correct?
> 
> The translated template files are copied into a de-DE directory in the
> plugin directory rather than being added to all the existing core translated
> files. This is so that if a plugin is ever uninstalled it removes the
> translated files and they're not hanging around with the core templates. We
> use the de-DE directory that now sits in the plugin to substitute the
> translated templates, the same way as we do in core Koha

I think it makes sense to store the translations within the plugin, but I assumed we'd use the "on the fly technique" that would not require creating a translated version of the template files.

This might give an idea of what I mean:
https://wiki.koha-community.org/wiki/Internationalization,_plural_forms,_context,_and_more_RFC

Or here: 
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=26392
 
> > In that case I am not sure why an install translation step for plugins would
> > be needed, as I assume they would come with all available languages po files
> > as part of their source. What is achieved by installing a language for a
> > plugin?
> 
> The plugins contain the po files but not the translated templates. The
> install step is needed to translate these using the core translation code,
> rather than needing to write translation code into the plugins themselves.
> It also takes any javascript strings and adds them to the existing
> javascript translations. (The plugin actually handles the translation step
> itself, see below)

See above - I think we could avoid creating the templates and saving us some steps in doing so. 

I was hesitant to use the method on "all of Koha" back then, mostly for concerns about performance and readability... but maybe plugins are the perfect use case here. I assume that the .po/.mo files we'd be dealing with would be quite small.

*** Bug 17489 has been marked as a duplicate of this bug. ***

Hmm, I don't see a reason not to support older translations as well as the new just-in-time syntax?

I think it would need more work to go backwards to not having specific template files per language.

(In reply to Martin Renvoize (ashimema) from comment #10)
> Hmm, I don't see a reason not to support older translations as well as the
> new just-in-time syntax?
> 
> I think it would need more work to go backwards to not having specific
> template files per language.
I can see at least one reason: it's more work to maintain two systems instead of one and the newer system can do what the older system does and more.

That being said, I believe there is currently no attempt at using the newer system with plugins, so it's hard to know how it would work and what it would mean for plugin authors and koha developers. I'd like to try (if I find the time...)

Comment on attachment 169561 [details] [review]
Bug 37472: Enable extraction of strings from plugins

Review of attachment 169561 [details] [review]:
-----------------------------------------------------------------

::: gulpfile.js
@@ +353,4 @@
>  function po_update_installer_marc21 ()  { return po_update_type('installer-MARC21') }
>  function po_update_installer_unimarc () { return po_update_type('installer-UNIMARC') }
>  
> +const PLUGINS_BASE = "/kohadevbox/plugins";

It should probably be fetched from $KOHA_CONF (which can contains multiple directories)

s/probably/definitely/ at comment #12