Bug 33134 - Add some missing languages
Summary: Add some missing languages
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Janusz Kaczmarek
QA Contact: Testopia
URL:
Keywords:
: 23693 (view as bug list)
Depends on:
Blocks:
 
Reported: 2023-03-03 23:33 UTC by Janusz Kaczmarek
Modified: 2024-04-23 15:08 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 33134: Add some 75 missing languages (81.33 KB, patch)
2023-03-03 23:58 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 33134: Add some 76 missing languages (82.47 KB, patch)
2023-03-04 14:10 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 33134: Add some 76 missing languages (82.51 KB, patch)
2023-04-17 19:51 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 33134: Add some 76 missing languages (92.22 KB, patch)
2024-04-12 10:02 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Janusz Kaczmarek 2023-03-03 23:33:12 UTC
When languages come into facets in ES, Koha should know the names for the codes used records.  Otherwise the three-letters language code is being displayed in the facet instead of name.  Koha now knows about 77 languages, many are still missing.

A patch will follow.
Comment 1 Janusz Kaczmarek 2023-03-03 23:58:44 UTC
Created attachment 147743 [details] [review]
Bug 33134: Add some 75 missing languages

When languages come into facets in ES, Koha should know
the names for the codes used records.  Otherwise
the three-letters language code is being displayed in
the facet instead of name.  Koha now knows about 77
languages, many are still missing.

The most missed 75 languages, in order of occurrences
in my data sets, are (MARC 21 codes): syr, chu, cop, arc,
ang, epo, syc, gez, sla, san, csb, akk, gmh, egy, wen, gle,
fro, rom, mac, mal, uga, mis, sem, sux, pli, bos, che, dum,
kaz, tib, enm, frm, mlt, got, nds, ira, lin, uzb, ave, cho,
jrb, bnt, pal, tut, goh, hsb, inc, mlg, roa, cel, sam, bem,
fur, gem, hit, lad, nah, ota, peo, pro, tat, wel, yor, bra,
den, hau, her, kas, mga, myn, nya, pra, que, tir, tuk.

Here, I do not put language codes which are extremely rare
in my data sets.

Subtags and names for language_subtag_registry are taken from
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry,
English names for language_descriptions.description - from
https://www.loc.gov/marc/languages/language_code.html, and Polish names -
from https://przepisy.bn.org.pl/aneksy#25-kody-jezykow---marc-21.

Test plan:
==========
1. Hava a ES installation with reasonable number of records,
   with a variety of languages in 008/35-39 not present in
   Koha (you can modify your record with codes from the list above).
   Set FacetMaxCount to a high number (like 500 or so,
   dependiing on the size of your data set).
2. Do a search, like '*a*'.
3. Controll the content of the Language faset -- you should
   see some bare codes instead of language names.
4. Apply the patch -- update the database.
5. Perform the same search.
6. You should see the names in place of the codes.
7. Optionally, you can activate the Polish translation and
   you should be able to see Polish names for the added languages.
Comment 2 Janusz Kaczmarek 2023-03-04 14:10:46 UTC
Created attachment 147758 [details] [review]
Bug 33134: Add some 76 missing languages

Bug 33134: Add some 76 missing languages

When languages come into facets in ES, Koha should know
the names for the codes used records.  Otherwise
the three-letters language code is being displayed in
the facet instead of name.  Koha now knows about 77
languages, many are still missing.

The most missed 76 languages, in order of occurrences
in my data sets, are (MARC 21 codes): grc, syr, chu, cop, arc,
ang, epo, syc, gez, sla, san, csb, akk, gmh, egy, wen, gle,
fro, rom, mac, mal, uga, mis, sem, sux, pli, bos, che, dum,
kaz, tib, enm, frm, mlt, got, nds, ira, lin, uzb, ave, cho,
jrb, bnt, pal, tut, goh, hsb, inc, mlg, roa, cel, sam, bem,
fur, gem, hit, lad, nah, ota, peo, pro, tat, wel, yor, bra,
den, hau, her, kas, mga, myn, nya, pra, que, tir, tuk.

Here, I do not put language codes which are extremely rare
in my data sets.

Subtags and names for language_subtag_registry are taken from
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry,
English names for language_descriptions.description - from
https://www.loc.gov/marc/languages/language_code.html, and Polish names -
from https://przepisy.bn.org.pl/aneksy#25-kody-jezykow---marc-21.

Test plan:
==========
1. Hava a ES installation with reasonable number of records,
   with a variety of languages in 008/35-39 not present in
   Koha (you can modify your record with codes from the list above).
   Set FacetMaxCount to a high number (like 500 or so,
   dependiing on the size of your data set).
2. Do a search, like '*a*'.
3. Controll the content of the Language faset -- you should
   see some bare codes instead of language names.
4. Apply the patch -- update the database.
5. Perform the same search.
6. You should see the names in place of the codes.
7. Optionally, you can activate the Polish translation and
   you should be able to see Polish names for the added languages.
Comment 3 Nick Clemens 2023-04-17 19:51:22 UTC
Created attachment 149776 [details] [review]
Bug 33134: Add some 76 missing languages

When languages come into facets in ES, Koha should know
the names for the codes used records.  Otherwise
the three-letters language code is being displayed in
the facet instead of name.  Koha now knows about 77
languages, many are still missing.

The most missed 76 languages, in order of occurrences
in my data sets, are (MARC 21 codes): grc, syr, chu, cop, arc,
ang, epo, syc, gez, sla, san, csb, akk, gmh, egy, wen, gle,
fro, rom, mac, mal, uga, mis, sem, sux, pli, bos, che, dum,
kaz, tib, enm, frm, mlt, got, nds, ira, lin, uzb, ave, cho,
jrb, bnt, pal, tut, goh, hsb, inc, mlg, roa, cel, sam, bem,
fur, gem, hit, lad, nah, ota, peo, pro, tat, wel, yor, bra,
den, hau, her, kas, mga, myn, nya, pra, que, tir, tuk.

Here, I do not put language codes which are extremely rare
in my data sets.

Subtags and names for language_subtag_registry are taken from
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry,
English names for language_descriptions.description - from
https://www.loc.gov/marc/languages/language_code.html, and Polish names -
from https://przepisy.bn.org.pl/aneksy#25-kody-jezykow---marc-21.

Test plan:
==========
1. Hava a ES installation with reasonable number of records,
   with a variety of languages in 008/35-39 not present in
   Koha (you can modify your record with codes from the list above).
   Set FacetMaxCount to a high number (like 500 or so,
   dependiing on the size of your data set).
2. Do a search, like '*a*'.
3. Controll the content of the Language faset -- you should
   see some bare codes instead of language names.
4. Apply the patch -- update the database.
5. Perform the same search.
6. You should see the names in place of the codes.
7. Optionally, you can activate the Polish translation and
   you should be able to see Polish names for the added languages.

Signed-off-by: Nick <nick@bywatersolutions.com>
Comment 4 Katrin Fischer 2023-04-22 17:44:46 UTC
I don't want to be the mean person here, but we are missing the languages name in their own language for all of these new languages. This really makes a difference in the language pull downs in the advanced searches in staff and OPAC. I think we should try and add as many as possible of these as well.

Another small thing: the .sql file used to be organized by languages, scripts and regions, but we kinda messed that up. Also there is no visual order to entries. (noting because I was trying to make sense of things)

What do you think?
Comment 5 Janusz Kaczmarek 2023-04-22 20:49:19 UTC
OK, I'm just speculating: Is somewhere a place where you could find the names of the ancient languages in the language itself?  What would be the point for this? 

I think the serious major problem now, with ES and the language facet, is that in a humanistic library the user is presented with meaningless code entries in the facet instead of language names, at least in English.

Do we really need the language name in the language itself?  for what purpose?  It is nice if it is present in the advanced search but if it is not--like classic Syriac for syc or classis Greek including koine for grc or other exotic languages we hold in our libraries--it is not a big problem.  At least much less important than syc and grc etc. as elements of the language facet, IMVHO.

Any idea how to solve this and move it forward so that this issue would not remain frozen for next months or years?
Comment 6 Katrin Fischer 2023-04-22 20:55:53 UTC
(In reply to Janusz Kaczmarek from comment #5)
> OK, I'm just speculating: Is somewhere a place where you could find the
> names of the ancient languages in the language itself?  What would be the
> point for this? 
> 
> I think the serious major problem now, with ES and the language facet, is
> that in a humanistic library the user is presented with meaningless code
> entries in the facet instead of language names, at least in English.
> 
> Do we really need the language name in the language itself?  for what
> purpose?  It is nice if it is present in the advanced search but if it is
> not--like classic Syriac for syc or classis Greek including koine for grc or
> other exotic languages we hold in our libraries--it is not a big problem. 
> At least much less important than syc and grc etc. as elements of the
> language facet, IMVHO.
> 
> Any idea how to solve this and move it forward so that this issue would not
> remain frozen for next months or years?

That's why I wrote this:
I think we should try and add _as many as possible_ of these as well. I am aware that for some it makes no sense.

I believe we are also missing some more common ones. I'll think on it.
Comment 7 Janusz Kaczmarek 2023-04-22 21:06:51 UTC
So maybe we should add all these: https://www.loc.gov/marc/languages/language_code.html in English and as many in other languages as we could find. 

BTW we already have the file MARC21Languages.xsl both for opac and bootstrap and the translations in many languages.

Shouldn't we just add them all to the database?  This would be an easy brute force solution but it would not require changing of the program logic...
Comment 8 Katrin Fischer 2023-12-12 08:46:02 UTC
I am sorry I didn't get back to this one to have another look yet. Happy to hand over for another opinion to QA team.
Comment 9 Lari Strand 2024-01-11 07:45:14 UTC
Some languages that do not show on facets/ are missing from language_rfc4646_to_iso639 and they don't even seem to have a two letter rfc4646 code, like afa/Afroasiatic languages. Where does Koha use the rfc4646 code?

I added a couple of missing ones to the rfc4646_to_iso639 table using the three letter iso639 codes for both rfc4646_subtag and iso639_2_code fields in the language_rfc4646_to_iso639 and added a local translation to language_descriptions and now the language shows translated instead of just a three letter code in the facets.

Shouldn't we at least try to add all the languages to the language_rfc4646_to_iso639 -table? Like, all 400+ of them (loc.gov). Sounds like a lot of work to check out if the language codes have a rfc4646 equivalent. If the languages are missing from that table, there no use having a local translation in your database since it can't be fetched.

Ps. We have now 116 language codes in language_rfc4646_to_iso639 and translated to Finnish + Swedish (language_descriptions) in Koha-Suomi. We use Elasticsearch.
Comment 10 Lari Strand 2024-01-11 07:58:27 UTC
"Some languages that do not show on facets/ are missing from language_rfc4646_to_iso639"

*Some languages that do not show on facets/advanced search language limits with their language description (but only as their 3 letter code instead) are missing from language_rfc4646_to_iso639.
Comment 11 Katrin Fischer 2024-04-12 08:24:13 UTC
I've talked this through with Janusz at the hackfest and we looked at various catalogs to check if the display was impacted by the missing descriptions and found it to be working well. Please QA :)
Comment 12 Janusz Kaczmarek 2024-04-12 10:02:58 UTC
Created attachment 164822 [details] [review]
Bug 33134: Add some 76 missing languages

When languages come into facets in ES, Koha should know
the names for the codes used records.  Otherwise
the three-letters language code is being displayed in
the facet instead of name.  Koha now knows about 77
languages, many are still missing.

The most missed 76 languages, in order of occurrences
in my data sets, are (MARC 21 codes): grc, syr, chu, cop, arc,
ang, epo, syc, gez, sla, san, csb, akk, gmh, egy, wen, gle,
fro, rom, mac, mal, uga, mis, sem, sux, pli, bos, che, dum,
kaz, tib, enm, frm, mlt, got, nds, ira, lin, uzb, ave, cho,
jrb, bnt, pal, tut, goh, hsb, inc, mlg, roa, cel, sam, bem,
fur, gem, hit, lad, nah, ota, peo, pro, tat, wel, yor, bra,
den, hau, her, kas, mga, myn, nya, pra, que, tir, tuk.

Plus: gwi, dgr, chp asked in bug 23693.

Here, I do not put language codes which are extremely rare
in my data sets.

Subtags and names for language_subtag_registry are taken from
https://www.iana.org/assignments/language-subtag-registry/language-subtag-registry,
English names for language_descriptions.description - from
https://www.loc.gov/marc/languages/language_code.html, and Polish names -
from https://przepisy.bn.org.pl/aneksy#25-kody-jezykow---marc-21.

Test plan:
==========
1. Hava a ES installation with reasonable number of records,
   with a variety of languages in 008/35-39 not present in
   Koha (you can modify your record with codes from the list above).
   Set FacetMaxCount to a high number (like 500 or so,
   dependiing on the size of your data set).
2. Do a search, like '*a*'.
3. Controll the content of the Language faset -- you should
   see some bare codes instead of language names.
4. Apply the patch -- update the database.
5. Perform the same search.
6. You should see the names in place of the codes.
7. Optionally, you can activate the Polish translation and
   you should be able to see Polish names for the added languages.

Signed-off-by: Nick <nick@bywatersolutions.com>
Comment 13 Janusz Kaczmarek 2024-04-12 10:05:16 UTC
*** Bug 23693 has been marked as a duplicate of this bug. ***