Bug 24975 - Refactor database translations
Summary: Refactor database translations
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: I18N/L10N (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Julian Maurice
QA Contact:
URL:
Keywords: rel_21_05_candidate, RM_priority
Depends on:
Blocks: 24977
  Show dependency treegraph
 
Reported: 2020-03-25 13:22 UTC by Julian Maurice
Modified: 2020-11-11 21:31 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 24975: Refactor database translations (132.18 KB, patch)
2020-03-25 13:24 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24975: Keep the translations, prevent duplicates, and other fixes (11.14 KB, patch)
2020-03-26 08:28 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24975: Refactor database translations (132.18 KB, patch)
2020-04-03 13:52 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 24975: Keep the translations, prevent duplicates, and other fixes (11.21 KB, patch)
2020-04-03 13:52 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 24975: Refactor database translations (132.56 KB, patch)
2020-04-07 12:37 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24975: Keep the translations, prevent duplicates, and other fixes (11.21 KB, patch)
2020-04-07 12:37 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 24975: Fix l10n_source definition in kohastructure.sql (1.17 KB, patch)
2020-04-10 09:32 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 24975: Fix syntax issues (2.69 KB, patch)
2020-04-10 11:40 UTC, Jonathan Druart
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2020-03-25 13:22:35 UTC
This is an attempt to make the process of translating database strings (like itemtype descriptions) easier for developers and translators.
Comment 1 Julian Maurice 2020-03-25 13:24:53 UTC
Created attachment 101703 [details] [review]
Bug 24975: Refactor database translations

This is an attempt to make the process of translating database strings
(like itemtype descriptions) easier for developers and translators.

For developers by reducing the amount of code needed to enable
translations for a new kind of object (authorised values, letters, news,
...).
For translators by providing a single entry point for all translatable
strings

Table `localization` is replaced by two new tables `l10n_source` and
`l10n_target`. `l10n_source` contains the original text while
`l10n_target` contains the corresponding translations

`l10n_source` is automatically updated whenever an itemtype is
inserted, updated, or deleted, so it is always up to date (this is done
in Koha::Schema::Result::ItemType).
It means that if you delete an itemtype, translations get deleted too.
It also means that if you change an itemtype's description, translations
are lost (not sure if this should be fixed, since if description
changes, translations should probably be changed too)

It adds a new subroutine in Koha::I18N, `db_t`, which returns the
translation for a given context and string, according to the current
language. Translations are stored in memcached (if enabled).

So instead of having to declare a new subroutine
Koha::ItemType::translated_description, you can write
    db_t('itemtype', $itemtype->description)

In templates you can write
    [% ItemTypes.t(itemtype.description) %]

There's also a new page, accessible from the administration main page,
"Localization". It displays all translatable strings and allow to
translate them for all enabled languages.

Test plan:
1. Before applying the patch, make sure you have translated some
   itemtypes. Translations should be kept by the updatedatabase script.
   Before applying the updatedatabase you may want to comment the
   'DROP TABLE localization' statement if you want to keep this table
2. Apply patch, run updatedatabase.pl, run update_dbix_class_files.pl
   and reload starman
3. Go to Admin » Localization, verify that you have your itemtypes and
   their translations.
4. Verify that the filter is working (it filters on the source column
   only).
5. Translate some strings by clicking on one of the 'Translate' link,
   entering some text and clicking on the 'Save' button. You can remove
   a translation by emptying the text box.
6. Add a new item type. Verify that it appears in the Localization page
7. Change the description of this item type. Verify that the new
   description appears in Localization page and that the old description
   is gone
8. Delete this item type and verify that its description no longer
   appears in Localization page
9. Navigate through Koha (intranet and opac) in multiple languages and
   verify that item types are translated (if not, please verify in
   master too, not all occurrences were translated)
Comment 2 Jonathan Druart 2020-03-25 13:32:01 UTC
That looks like a great idea.
What about the interface (for AVs)?
Comment 3 Julian Maurice 2020-03-26 08:28:08 UTC
Created attachment 101786 [details] [review]
Bug 24975: Keep the translations, prevent duplicates, and other fixes

Translations are no longer lost when you change an itemtype's
description, thanks to a new `l10n_source.key` column which must be
unique within it's "text group" ('itemtype' is a text group)
This also prevent duplicate entries in l10n_source

Columns were renamed to better match their purpose:
* context -> group
* source -> text

Use Koha::Cache::Memory::Lite if other caching systems are not available

Prepend 'l10n:' to the cache key to avoid conflicts

Test plan:
1. DROP TABLE IF EXISTS l10n_target, l10n_source
2. Follow the test plan from previous patch
Comment 4 Martin Renvoize 2020-03-26 12:10:57 UTC
Thanks for picking this up Julian, it's looking great so far! :)
Comment 5 Martin Renvoize 2020-04-03 13:52:33 UTC
Created attachment 102387 [details] [review]
Bug 24975: Refactor database translations

This is an attempt to make the process of translating database strings
(like itemtype descriptions) easier for developers and translators.

For developers by reducing the amount of code needed to enable
translations for a new kind of object (authorised values, letters, news,
...).
For translators by providing a single entry point for all translatable
strings

Table `localization` is replaced by two new tables `l10n_source` and
`l10n_target`. `l10n_source` contains the original text while
`l10n_target` contains the corresponding translations

`l10n_source` is automatically updated whenever an itemtype is
inserted, updated, or deleted, so it is always up to date (this is done
in Koha::Schema::Result::ItemType).
It means that if you delete an itemtype, translations get deleted too.
It also means that if you change an itemtype's description, translations
are lost (not sure if this should be fixed, since if description
changes, translations should probably be changed too)

It adds a new subroutine in Koha::I18N, `db_t`, which returns the
translation for a given context and string, according to the current
language. Translations are stored in memcached (if enabled).

So instead of having to declare a new subroutine
Koha::ItemType::translated_description, you can write
    db_t('itemtype', $itemtype->description)

In templates you can write
    [% ItemTypes.t(itemtype.description) %]

There's also a new page, accessible from the administration main page,
"Localization". It displays all translatable strings and allow to
translate them for all enabled languages.

Test plan:
1. Before applying the patch, make sure you have translated some
   itemtypes. Translations should be kept by the updatedatabase script.
   Before applying the updatedatabase you may want to comment the
   'DROP TABLE localization' statement if you want to keep this table
2. Apply patch, run updatedatabase.pl, run update_dbix_class_files.pl
   and reload starman
3. Go to Admin » Localization, verify that you have your itemtypes and
   their translations.
4. Verify that the filter is working (it filters on the source column
   only).
5. Translate some strings by clicking on one of the 'Translate' link,
   entering some text and clicking on the 'Save' button. You can remove
   a translation by emptying the text box.
6. Add a new item type. Verify that it appears in the Localization page
7. Change the description of this item type. Verify that the new
   description appears in Localization page and that the old description
   is gone
8. Delete this item type and verify that its description no longer
   appears in Localization page
9. Navigate through Koha (intranet and opac) in multiple languages and
   verify that item types are translated (if not, please verify in
   master too, not all occurrences were translated)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 6 Martin Renvoize 2020-04-03 13:52:36 UTC
Created attachment 102388 [details] [review]
Bug 24975: Keep the translations, prevent duplicates, and other fixes

Translations are no longer lost when you change an itemtype's
description, thanks to a new `l10n_source.key` column which must be
unique within it's "text group" ('itemtype' is a text group)
This also prevent duplicate entries in l10n_source

Columns were renamed to better match their purpose:
* context -> group
* source -> text

Use Koha::Cache::Memory::Lite if other caching systems are not available

Prepend 'l10n:' to the cache key to avoid conflicts

Test plan:
1. DROP TABLE IF EXISTS l10n_target, l10n_source
2. Follow the test plan from previous patch

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2020-04-03 14:02:23 UTC
I like this, and it's working pretty well in testing.. few comments however

1/ The 'Localization' option appears in the admin home screen whether or not you have any translations installed.. if you then navigate into it you get an ajax request that fails for the datatable if there are no translations available.
2/ You switch from Koha::Object classes to plain DBIx ResultSets.. was that deliberate and if so can you explain why (I'm betting it was a performance perspective?)
Comment 8 Martin Renvoize 2020-04-03 14:44:19 UTC
(In reply to Martin Renvoize from comment #7)
> I like this, and it's working pretty well in testing.. few comments however
> 
> 1/ The 'Localization' option appears in the admin home screen whether or not
> you have any translations installed.. if you then navigate into it you get
> an ajax request that fails for the datatable if there are no translations
> available.
> 2/ You switch from Koha::Object classes to plain DBIx ResultSets.. was that
> deliberate and if so can you explain why (I'm betting it was a performance
> perspective?)

Both points answered on IRC.. 1/ was a mistake my end.. forgot to rebuild the schema's after applying the patches on second test run.

Signing off.
Comment 9 Julian Maurice 2020-04-03 14:46:55 UTC
(In reply to Martin Renvoize from comment #7)
> 2/ You switch from Koha::Object classes to plain DBIx ResultSets.. was that
> deliberate and if so can you explain why (I'm betting it was a performance
> perspective?)

(pasting my answer on IRC here for reference:)
I had performance in mind (but i did not measure it), but also, some parts of Koha use DBIx::Class directly, so by putting the code at DBIX::Class level it should work whether you use dbix::class or koha::objects.

Thanks for testing :)
Comment 10 Jonathan Druart 2020-04-07 12:37:13 UTC
Created attachment 102513 [details] [review]
Bug 24975: Refactor database translations

This is an attempt to make the process of translating database strings
(like itemtype descriptions) easier for developers and translators.

For developers by reducing the amount of code needed to enable
translations for a new kind of object (authorised values, letters, news,
...).
For translators by providing a single entry point for all translatable
strings

Table `localization` is replaced by two new tables `l10n_source` and
`l10n_target`. `l10n_source` contains the original text while
`l10n_target` contains the corresponding translations

`l10n_source` is automatically updated whenever an itemtype is
inserted, updated, or deleted, so it is always up to date (this is done
in Koha::Schema::Result::ItemType).
It means that if you delete an itemtype, translations get deleted too.
It also means that if you change an itemtype's description, translations
are lost (not sure if this should be fixed, since if description
changes, translations should probably be changed too)

It adds a new subroutine in Koha::I18N, `db_t`, which returns the
translation for a given context and string, according to the current
language. Translations are stored in memcached (if enabled).

So instead of having to declare a new subroutine
Koha::ItemType::translated_description, you can write
    db_t('itemtype', $itemtype->description)

In templates you can write
    [% ItemTypes.t(itemtype.description) %]

There's also a new page, accessible from the administration main page,
"Localization". It displays all translatable strings and allow to
translate them for all enabled languages.

Test plan:
1. Before applying the patch, make sure you have translated some
   itemtypes. Translations should be kept by the updatedatabase script.
   Before applying the updatedatabase you may want to comment the
   'DROP TABLE localization' statement if you want to keep this table
2. Apply patch, run updatedatabase.pl, run update_dbix_class_files.pl
   and reload starman
3. Go to Admin » Localization, verify that you have your itemtypes and
   their translations.
4. Verify that the filter is working (it filters on the source column
   only).
5. Translate some strings by clicking on one of the 'Translate' link,
   entering some text and clicking on the 'Save' button. You can remove
   a translation by emptying the text box.
6. Add a new item type. Verify that it appears in the Localization page
7. Change the description of this item type. Verify that the new
   description appears in Localization page and that the old description
   is gone
8. Delete this item type and verify that its description no longer
   appears in Localization page
9. Navigate through Koha (intranet and opac) in multiple languages and
   verify that item types are translated (if not, please verify in
   master too, not all occurrences were translated)

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 11 Jonathan Druart 2020-04-07 12:37:16 UTC
Created attachment 102514 [details] [review]
Bug 24975: Keep the translations, prevent duplicates, and other fixes

Translations are no longer lost when you change an itemtype's
description, thanks to a new `l10n_source.key` column which must be
unique within it's "text group" ('itemtype' is a text group)
This also prevent duplicate entries in l10n_source

Columns were renamed to better match their purpose:
* context -> group
* source -> text

Use Koha::Cache::Memory::Lite if other caching systems are not available

Prepend 'l10n:' to the cache key to avoid conflicts

Test plan:
1. DROP TABLE IF EXISTS l10n_target, l10n_source
2. Follow the test plan from previous patch

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 12 Jonathan Druart 2020-04-07 12:38:43 UTC
It does not work for me. The table does not load on admin/localization.pl

DBIx::Class::Storage::DBI::_dbh_execute(): Unknown column 'me.context' in 'field list' at /kohadevbox/koha/svc/localization line 35

I rebased the patches because of conflicts with bug 4944.
Comment 13 Julian Maurice 2020-04-10 08:56:00 UTC
(In reply to Jonathan Druart from comment #12)
> It does not work for me. The table does not load on admin/localization.pl
> 
> DBIx::Class::Storage::DBI::_dbh_execute(): Unknown column 'me.context' in
> 'field list' at /kohadevbox/koha/svc/localization line 35

You should probably recreate the DBIx schema files, 'context' has been renamed 'group' in the last patch.
Comment 14 Julian Maurice 2020-04-10 09:32:18 UTC
Created attachment 102688 [details] [review]
Bug 24975: Fix l10n_source definition in kohastructure.sql
Comment 15 Jonathan Druart 2020-04-10 11:40:11 UTC
Created attachment 102701 [details] [review]
Bug 24975: Fix syntax issues
Comment 16 Jonathan Druart 2020-04-10 12:33:46 UTC
I've talked a bit with Julian, and there is something that is not correct IMO.

The translation for the item type "BK/Books" is picked using its description, "Books". Which means that if we have another itemtype with the same description "BK2/Books", its translation string will be used for BK.

For instance:
 BK/Books translated in fr-FR => "Livre"
 BK2/Books translated in fr-FR => "Autre livre"

Item type BK will be display "Autre livre" in French.

I should also note that the collation for the 2 new tables is utf8mb4_bin (instead of the usual utf8mb4_unicode_ci), to differentiate "Book" and "book".
But I have tried:
 BK/Books translated in fr-FR => "Livre"
 BK2/books translated in fr-FR => "Autre livre"
An I still see "Autre livre" for "BK/Books".

I know it does not make a lot a sense in a real-life situation, but I think we hit a design issue. This problem might be highlighted even more for AV, when we will need to take into account the "interface".

I think we should rethink this a bit and have (add back) a method at Koha::ItemType level to retrieve the translated string. That will give us more flexibility for future changes (now we need to update all the occurrences of db_t).

I still think this implementation is better than the original one, but it still need a bit of tuning to be ready IMO.


Failing QA/In Discussion.
Comment 17 Julian Maurice 2020-04-10 14:51:20 UTC
> I still think this implementation is better than the original one, but
> it still need a bit of tuning to be ready IMO.

I think there is no perfect solution, but maybe the best solution, and
the simplest one, is to go back to using the code of itemtype/AV,
instead of the description, like you suggested.
With codes there will be no duplicates, and no collation issues.
Comment 18 Martin Renvoize 2020-04-16 07:48:26 UTC
Are you hoping to continue here Julian.. it's looking so promising :)
Comment 19 Julian Maurice 2020-04-16 10:07:08 UTC
(In reply to Martin Renvoize from comment #18)
> Are you hoping to continue here Julian.. it's looking so promising :)

Yes. I don't know when but I will continue
Comment 20 Julian Maurice 2020-09-04 13:10:08 UTC
I've been rethinking about this.

If we fetch a translation using a key like "itemtype-BK", or "authorisedvalue-CCODE-FIC-OPAC" it might work well for these two type of entities, because there would be a 1:1 relationship between every string and its translation, and there should not be too much duplicated translations.
But what if we want to make MARC frameworks translatable too ? Say I have 10 frameworks that share 95% of their field/subfield descriptions, I'll have to translate each of these strings 10 times, because there would be a translation for "framework-FW1-245a-OPAC" and another for "framework-FW2-245a-OPAC" and so on. Waste of time and DB space.

So I think it would be better to fetch a translation using the text, for instance "framework-Title". That way every field/subfield description in a MARC framework that is "Title" would be translated the same way and the translator would need to translate it only once. Bonus: if you create a new framework it will already be translated.

Now there is still a design issue with the current implementation (comment 16) where Koha allows to give multiple translations to the same original text, but only one of those will be picked and the others will never show up. I think it can be solved by using a different db schema, but before going into more details I would like to hear what people think about the "key vs text" topic.

By the way, I think the collation issue in comment 16 was in fact a cache issue. The cache is not always cleared when it should be, and I tested the exact same example and it worked for me.
Comment 21 Martin Renvoize 2020-09-04 16:25:25 UTC
(In reply to Julian Maurice from comment #20)
> I've been rethinking about this.
> 
> If we fetch a translation using a key like "itemtype-BK", or
> "authorisedvalue-CCODE-FIC-OPAC" it might work well for these two type of
> entities, because there would be a 1:1 relationship between every string and
> its translation, and there should not be too much duplicated translations.
> But what if we want to make MARC frameworks translatable too ? Say I have 10
> frameworks that share 95% of their field/subfield descriptions, I'll have to
> translate each of these strings 10 times, because there would be a
> translation for "framework-FW1-245a-OPAC" and another for
> "framework-FW2-245a-OPAC" and so on. Waste of time and DB space.
> 
> So I think it would be better to fetch a translation using the text, for
> instance "framework-Title". That way every field/subfield description in a
> MARC framework that is "Title" would be translated the same way and the
> translator would need to translate it only once. Bonus: if you create a new
> framework it will already be translated.
> 
> Now there is still a design issue with the current implementation (comment
> 16) where Koha allows to give multiple translations to the same original
> text, but only one of those will be picked and the others will never show
> up. I think it can be solved by using a different db schema, but before
> going into more details I would like to hear what people think about the
> "key vs text" topic.
> 
> By the way, I think the collation issue in comment 16 was in fact a cache
> issue. The cache is not always cleared when it should be, and I tested the
> exact same example and it worked for me.

I'm confused.. you seem to suggest shifting from code based keys to description based keys and then revert again in the next paragraph.

I feel like we should be using code based keys for flexability/reliability but allow for a fallback chain for cases like frameworks..  so for example 'framework-FW2-245a-OPAC' could fallback to a 'framework-*-245a-OPAC' for a case where the FW2 specific case doesn't exist?

I need to look at the code again to really get to grips with that suggestion so please feel free to cast it aside if it's non-sense.
Comment 22 Julian Maurice 2020-09-04 16:35:22 UTC
(In reply to Martin Renvoize from comment #21)
> (In reply to Julian Maurice from comment #20)
> > I've been rethinking about this.
> > 
> > If we fetch a translation using a key like "itemtype-BK", or
> > "authorisedvalue-CCODE-FIC-OPAC" it might work well for these two type of
> > entities, because there would be a 1:1 relationship between every string and
> > its translation, and there should not be too much duplicated translations.
> > But what if we want to make MARC frameworks translatable too ? Say I have 10
> > frameworks that share 95% of their field/subfield descriptions, I'll have to
> > translate each of these strings 10 times, because there would be a
> > translation for "framework-FW1-245a-OPAC" and another for
> > "framework-FW2-245a-OPAC" and so on. Waste of time and DB space.
> > 
> > So I think it would be better to fetch a translation using the text, for
> > instance "framework-Title". That way every field/subfield description in a
> > MARC framework that is "Title" would be translated the same way and the
> > translator would need to translate it only once. Bonus: if you create a new
> > framework it will already be translated.
> > 
> > Now there is still a design issue with the current implementation (comment
> > 16) where Koha allows to give multiple translations to the same original
> > text, but only one of those will be picked and the others will never show
> > up. I think it can be solved by using a different db schema, but before
> > going into more details I would like to hear what people think about the
> > "key vs text" topic.
> > 
> > By the way, I think the collation issue in comment 16 was in fact a cache
> > issue. The cache is not always cleared when it should be, and I tested the
> > exact same example and it worked for me.
> 
> I'm confused.. you seem to suggest shifting from code based keys to
> description based keys and then revert again in the next paragraph.
Sorry for the confusion... where did I lost you exactly ? :)
Just to be clear, I suggest description based keys.

> I feel like we should be using code based keys for flexability/reliability
> but allow for a fallback chain for cases like frameworks..  so for example
> 'framework-FW2-245a-OPAC' could fallback to a 'framework-*-245a-OPAC' for a
> case where the FW2 specific case doesn't exist?
But what if the description of 245a for FW2 is different from the 245a of default framework ? Won't we have a wrong translation ?
Comment 23 Martin Renvoize 2020-09-14 13:18:07 UTC
I'm coming back around to keying on the original text now rather than a definitive unique key.

However, I think we should allow for the key to be passed in additional and perhaps other variables too eventually.

My thinking behind this is all the various options we have now for plural forms and the likes in the rest of the translation system. It's those cases I could see requiring additional options to refine which version of a translation to 'pick'.

So.. my thinking is along the lines

db_t($itype->itemtype, $itype->description, { key => $itype->code, count => $x });

Where the first two params are required, but the hashref or additional arguments is optional depending on the situation.

This way, we can key on the description text and allow for passing the code as an option to allow for more specific selection it it exists but fallback to just description as the key if it does not. Something similar to the way we allow falling back through rules in the circ rules system.. if the more specific case doesn't exist we trap the less specific case.

I'll have a little play with the code but I think this could end up flexible enough to support future expansions and allow for more and less specific cases in translation as required.
Comment 24 Jonathan Druart 2020-09-28 09:18:13 UTC
Katrin, Bernardo, can you give us your feeling on this topic please?
Comment 25 Martin Renvoize 2020-09-28 16:57:21 UTC
Code speaks more than words.. I've been having a play with a couple of followups to use a 'key' rather than rely on the string as the key for lookup, adding 'fuzzy' support and re-introducing an accessor method on the Koha::ItemType object:

https://gitlab.com/mrenvoize/Koha/-/commits/bug_24975

Still a work in progress but would love some feedback as to whether this might be a route to adopt.
Comment 26 Julian Maurice 2020-09-29 07:25:37 UTC
(In reply to Martin Renvoize from comment #25)
> Code speaks more than words.. I've been having a play with a couple of
> followups to use a 'key' rather than rely on the string as the key for
> lookup, adding 'fuzzy' support and re-introducing an accessor method on the
> Koha::ItemType object:
> 
> https://gitlab.com/mrenvoize/Koha/-/commits/bug_24975
> 
> Still a work in progress but would love some feedback as to whether this
> might be a route to adopt.

Does it solve the problem we mentioned with framework translations ?
Comment 27 Martin Renvoize 2020-09-29 07:36:05 UTC
We key on new 'key' field. So it leaves it up to the person whome implements the next translatable feature to pick an appropriate key.

For frameworks I would suggest a group of 'framework' and a key of 'marc field'.. thus allowing to translate at the default framework level but keep marc fields distinct for the cases as mentioned by Jonas on IRC.
Comment 28 Martin Renvoize 2020-09-29 08:03:39 UTC
What I'm also tempted to do is add a standardly named accessor be to objects with translatable fields.. ->translation_key which would produce the right key from the internal fields behind the scenes. So for itemtype it simply passes through itemtype, but for a it might pass $code_$interface for example.
Comment 29 Julian Maurice 2020-09-29 08:15:15 UTC
(In reply to Martin Renvoize from comment #27)
> We key on new 'key' field. So it leaves it up to the person whome implements
> the next translatable feature to pick an appropriate key.
> 
> For frameworks I would suggest a group of 'framework' and a key of 'marc
> field'.. thus allowing to translate at the default framework level but keep
> marc fields distinct for the cases as mentioned by Jonas on IRC.

I'm confused. Do you mean something like
  db_t('framework', 'marc field', { marc_field => '245a', type => 'opac' })
or
  db_t('framework', '245a-opac')
or
  db_t('framework-FW1', '245a-opac')
or something else ?

I could not find what Jonas mentioned on IRC, that probably doesn't help me to understand :) Can you paste it or summarize it here ?
Comment 30 Martin Renvoize 2020-09-29 08:41:29 UTC
(In reply to Julian Maurice from comment #29)
> (In reply to Martin Renvoize from comment #27)
> > We key on new 'key' field. So it leaves it up to the person whome implements
> > the next translatable feature to pick an appropriate key.
> > 
> > For frameworks I would suggest a group of 'framework' and a key of 'marc
> > field'.. thus allowing to translate at the default framework level but keep
> > marc fields distinct for the cases as mentioned by Jonas on IRC.
> 
> I'm confused. Do you mean something like
>   db_t('framework', 'marc field', { marc_field => '245a', type => 'opac' })
> or
>   db_t('framework', '245a-opac')
> or
>   db_t('framework-FW1', '245a-opac')
> or something else ?
> 
> I could not find what Jonas mentioned on IRC, that probably doesn't help me
> to understand :) Can you paste it or summarize it here ?

db_t('framework', '245a');

or if you need it at an interface level too

db_t('framework', '245a-opac');

i.e. We leave defining the granularity of the key to the developer adding the translation option to the component and we discus that granularity there.

What I've not done is any form of fallback system.. i.e. if we wanted to allow one to optionally define translations at a more granular level, for per framework for example.

example

db_t('framework', 'FA-245a');.. fall back to just '245a' if the 'FA-' translation doesn't exist.  I felt that was out of scope for here.. and perhaps not really required.

I'll go dig out the IRC log.
Comment 31 Martin Renvoize 2020-09-29 08:45:29 UTC
>kohaputti
maybe 013e and 583l could use different translations in finnish, 583l = "Kunto" and 013l = "Tila"
I mean you could translate Kunto also as Tila I guess
so not the best example but someone could do it like this...
>kohaputti
a safe bet here would be frameworks+field+subfield, the meaning cannot change between different frameworks, can it? Also, why even allow changing the descriptions for each framework separately
Comment 32 Martin Renvoize 2020-09-29 08:46:14 UTC
To add detail to that.. it's two different translations for the single word 'Status' in English for the Finnish translation.
Comment 33 Julian Maurice 2020-09-29 08:50:37 UTC
(In reply to Martin Renvoize from comment #30)
> db_t('framework', 'FA-245a');.. fall back to just '245a' if the 'FA-'
> translation doesn't exist.  I felt that was out of scope for here.. and
> perhaps not really required.

The problem with that is that we don't know if we should fall back. 245a may have a completely different description in FA than in default framework. (Well maybe not for 245a, but you get the point)
Comment 34 Martin Renvoize 2020-09-29 08:55:00 UTC
My point is.. I've tried to make it flexible enough without introducing too much additional overhead/code complexity here. I suggest we should not second guess the level of translation requirements/granularity of different components here but rather defer that to each bug introducing additional translation options.
Comment 35 Julian Maurice 2020-09-29 11:53:16 UTC
(In reply to Martin Renvoize from comment #28)
> What I'm also tempted to do is add a standardly named accessor be to objects
> with translatable fields.. ->translation_key which would produce the right
> key from the internal fields behind the scenes. So for itemtype it simply
> passes through itemtype, but for a it might pass $code_$interface for
> example.

That means Koha::Object(s) should be aware of the interface (opac/staff) ? :/

(In reply to Martin Renvoize from comment #34)
> My point is.. I've tried to make it flexible enough without introducing too
> much additional overhead/code complexity here. I suggest we should not
> second guess the level of translation requirements/granularity of different
> components here but rather defer that to each bug introducing additional
> translation options.
We have the same goal, just different opinions on how to achieve that. If we do it right from the beginning, making more kind of objects translatable will be easier. I think it's worth taking some time to think about it.

By using codes as translation key, it can work for MARC frameworks too, but it will require to translate the same strings multiple times. To avoid that, we would need to implement some kind of fallback mechanism.
By using descriptions as translation key, there is no need for fallback. Every string have to be translated only once. However it doesn't allow to translate the same string in different ways (like in comment 31). Doing that will also require a fallback mechanism.

So, do we want to use a code as key (eg. "245a") and refine with other codes ("FW2"),
or do we want to use a description as key (eg. "Title") and refine with other codes ("FW2", "245a") ?
I'd say the fallback for a description-based key is more likely to be accurate ("Kunto" is not very different in meaning from "Tila"), whereas a fallback for a code-based key can be simply wrong if descriptions are different.

Now that I think about it, this refinement system could also include the group ("itemtype", "framework", ...) so we'd have to translate "Title" only once, and it would be translated everywhere the same way, except if one add a different translation for [framework, FW2, 245a] for instance.

That being said, I recognize that frameworks are a particular case, and it may not be worth it to add all these translation features just for that. But having a separate translation system for frameworks is not ideal either... It would be simpler if all frameworks shared their descriptions :)

BTW, I really like the idea of "fuzzy" you've added
Comment 36 Jonathan Druart 2020-11-11 08:17:47 UTC
Could we get a work team on this one and target 21.05?
Comment 37 Katrin Fischer 2020-11-11 21:31:46 UTC
Also very interested in this, can provide testing and QA. A quick discussion on how to approach this would be nice!