Bug 15067

Summary: Add additional languages to advanced search language search
Product: Koha Reporter: Katrin Fischer <katrin.fischer>
Component: OPACAssignee: Ethan <ethan.amohia>
Status: CLOSED FIXED QA Contact: Nick Clemens <nick>
Severity: enhancement    
Priority: P5 - low CC: chloealabaster, chris, danielle, david, dcook, egpetridis, ellermaugustus, ethan.amohia, fridolin.somers, jonathan.druart, kyle, matthias.meusburger, mtj, nazrul.it.puc, nick, tomascohen, veron
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23693
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
This enhancement adds Estonian, Inuktitut, Inupiaq, Latvian, and Lithuanian (along with their translations) to the list of languages in the advanced search for the OPAC and staff interface (Advanced search > More options > Language drop down list). The list of languages is also now sorted in alphabetical order.
Version(s) released in:
21.11.00
Bug Depends on:    
Bug Blocks: 18493, 29605, 29631    
Attachments: 15067 - Add additional languages to advanced search language search
15067 - Add additional languages to advanced search language search
Bug 15067 - Follow up to fix sorting
[SIGNED-OFF] Bug 15067 - Follow up to fix sorting
[SIGNED-OFF] 15067 - Add additional languages to advanced search language search
[SIGNED-OFF] Bug 15067 - Follow up to fix sorting
Bug 15067: Added translations of languages to those 5 languages.
Bug 15067: Added translations of languages to those 5 languages.
Bug 15067: Added translations of languages to those 5 languages.
[SIGNED OFF] Bug 15067: Added translations of languages to those 5 languages
Bug 15067: Follow up to fix sorting
Bug 15067: Add additional languages to advanced search language search
Bug 15067: Added translations of languages to those 5 languages
Bug 15067: Follow up to fix sorting
Bug 15067: Add additional languages to advanced search language search
Bug 15067: Added translations of languages to those 5 languages
Bug 15067: Follow up to fix sorting
Bug 15067: Add additional languages to advanced search language search
Bug 15067: Added translations of languages to those 5 languages
Bug 15067: (QA follow-up) Ensure update is idempotent
Bug 15067: (QA follow-up) Add missing UNIQUE constraint in kohastructure.sql
Bug 15067: (QA follow-up) Add missing UNIQUE constraint in kohastructure.sql

Description Katrin Fischer 2015-10-27 10:34:25 UTC
The language option list in the advanced search can now be configured now, but not all languages are available. 3 of the missing that should be added are:

Estonian
Latvian
Lithuanian
Comment 1 Danielle Elder 2016-01-11 16:07:14 UTC
Other languages missing are Inuktitut (iku) and Inupiaq (ipk).
Comment 2 Chloe Alabaster 2016-01-20 02:54:54 UTC
looks like it needs to be added onto multiple database tables, might try again tomorrow
Comment 3 Gus Ellerm 2016-01-21 02:28:43 UTC Comment hidden (obsolete)
Comment 4 Chloe Alabaster 2016-01-21 23:04:20 UTC
I tested this, cant sign off but it worked for me

--as a side note it would be nice if the languages were sorted in alphabetical order to that it is easier to find the language you want..at the moment they seem to be sorted by when they were added to the database eg the ones Gus has added are the last languages on there.
Comment 5 Nick Clemens 2016-01-25 20:47:11 UTC Comment hidden (obsolete)
Comment 6 Nick Clemens 2016-01-25 20:49:22 UTC Comment hidden (obsolete)
Comment 7 Aleisha Amohia 2016-01-26 01:03:37 UTC Comment hidden (obsolete)
Comment 8 Aleisha Amohia 2016-01-26 01:05:28 UTC
Created attachment 47286 [details] [review]
[SIGNED-OFF] 15067 - Add additional languages to advanced search language search

run update database script, look for new laguages under 'Language:' in Advanced search.
Languages that should be added are:
1. Estonian
2. Latvian
3. Lithuanian
4. Inuktitut
5. Inupiaq

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 9 Aleisha Amohia 2016-01-26 01:05:40 UTC
Created attachment 47287 [details] [review]
[SIGNED-OFF] Bug 15067 - Follow up to fix sorting

It looks like C4::Languages:getLanguages relies on implicit database
order for languages. They appear to be sorted by subtag so this patch
adds that sorting to the database query to ensure the dropdown is
sorted.

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Comment 10 Jonathan Druart 2016-01-27 12:07:58 UTC
I'd say this patch is not complete.

Looking at the other part of the files, we have, for instance:

  INSERT INTO language_descriptions(subtag, type, lang, description)
  VALUES ( 'fi', 'language', 'fi', 'Suomi');

  INSERT INTO language_descriptions(subtag, type, lang, description)
  VALUES ( 'fi', 'language', 'en', 'Finnish');

We need the language name to be translated in its own language

Moreover it seems that we need to add the country with the "region" type too.
Comment 11 Ethan 2019-01-17 04:03:26 UTC
Created attachment 84122 [details] [review]
Bug 15067: Added translations of languages to those 5 languages.

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.
Comment 12 Ethan 2019-01-17 04:12:17 UTC
Created attachment 84124 [details] [review]
Bug 15067: Added translations of languages to those 5 languages.

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.

Test Plan:
1. From homepage, go to Advanced search
2. Click on More options next to the Search button
3. Scroll down to the Limits section
4. Click on Language: No limit
5. Find Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq at the
bottom of the dropdown.
6. They should all be there.
7. Estonian and Latvian are repeated, this need to be fixed.
8. They need to be put in alphabetical order.
Comment 13 Hayley Pelham 2019-03-06 00:32:10 UTC
Created attachment 86122 [details] [review]
Bug 15067: Added translations of languages to those 5 languages.

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.

Test Plan:
1. From homepage, go to Advanced search
2. Click on More options next to the Search button
3. Scroll down to the Limits section
4. Click on Language: No limit
5. Find Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq at the
bottom of the dropdown.
6. They should all be there.
7. Estonian and Latvian are repeated, this need to be fixed.
8. They need to be put in alphabetical order.

Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
The five languages are present, sorted alphabetically, and are not
repeated.
Comment 14 Katrin Fischer 2019-04-09 10:49:26 UTC
Hi Hayley, 

some unwanted changes have slipped into the last patch:

 Koha/Illbackends/BLDSS                                |  1 +
 Koha/Illbackends/koha-ill-freeform                    |  1 +
 .../mysql/atomicupdate/bug_15067-add_newlanguages.sql | 19 +++++++++++++++++++
 installer/data/mysql/mandatory/subtag_registry.sql    | 15 +++++++++++++++
 koha-ill-freeform                                     |  1 +
 5 files changed, 37 insertions(+)
 create mode 160000 Koha/Illbackends/BLDSS
 create mode 160000 Koha/Illbackends/koha-ill-freeform
 create mode 160000 koha-ill-freeform

Can you please fix the patch and switch back to signed off?
Comment 15 Hayley Pelham 2019-05-12 22:12:44 UTC
Created attachment 89642 [details] [review]
[SIGNED OFF] Bug 15067: Added translations of languages to those 5 languages

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.

Test Plan:
1. From homepage, go to Advanced search
2. Click on More options next to the Search button
3. Scroll down to the Limits section
4. Click on Language: No limit
5. Find Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq at the
bottom of the dropdown.
6. They should all be there.
7. Estonian and Latvian are repeated, this need to be fixed.
8. They need to be put in alphabetical order.

Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
The five languages are present, sorted alphabetically, and are not
repeated.
Comment 16 Katrin Fischer 2019-05-13 19:14:41 UTC
Hi Hayley,

wouldn't make ordering by description make more sense? The codes might not reflect the translation, making the list appear unordered again.

There is also a problem found by the QA tools:

 FAIL	C4/Languages.pm
   OK	  critic
   OK	  forbidden patterns
   OK	  git manipulation
   FAIL	  pod
		*** ERROR: 
		Spurious =cut command
		 
		 in file C4/Languages.pm

Can you please check?
Comment 17 Matthias Meusburger 2021-05-20 12:56:45 UTC
Created attachment 121239 [details] [review]
Bug 15067: Follow up to fix sorting

It looks like C4::Languages:getLanguages relies on implicit database
order for languages. They appear to be sorted by subtag so this patch
adds that sorting to the database query to ensure the dropdown is
sorted.

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>
Comment 18 Matthias Meusburger 2021-05-20 12:57:44 UTC
Created attachment 121240 [details] [review]
Bug 15067: Add additional languages to advanced search language search

run update database script, look for new laguages under 'Language:' in Advanced search.
Languages that should be added are:
1. Estonian
2. Latvian
3. Lithuanian
4. Inuktitut
5. Inupiaq

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 19 Matthias Meusburger 2021-05-20 13:02:06 UTC
Created attachment 121241 [details] [review]
Bug 15067: Added translations of languages to those 5 languages

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.

Test Plan:
1. From homepage, go to Advanced search
2. Click on More options next to the Search button
3. Scroll down to the Limits section
4. Click on Language: No limit
5. Find Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq at the
bottom of the dropdown.
6. They should all be there.
7. Estonian and Latvian are repeated, this need to be fixed.
8. They need to be put in alphabetical order.

Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
The five languages are present, sorted alphabetically, and are not
repeated.
Comment 20 Matthias Meusburger 2021-05-20 13:05:37 UTC
Hi,

Sorting is now done on description.

Also, QA tools is now ok:

koha-qa.pl -c 3 -v 2

 OK	C4/Languages.pm
   OK	  critic
   OK	  forbidden patterns
   OK	  git manipulation
   OK	  pod
   OK	  pod coverage
   OK	  spelling
   OK	  valid
Comment 21 David Nind 2021-05-20 21:13:42 UTC
Created attachment 121260 [details] [review]
Bug 15067: Follow up to fix sorting

It looks like C4::Languages:getLanguages relies on implicit database
order for languages. They appear to be sorted by subtag so this patch
adds that sorting to the database query to ensure the dropdown is
sorted.

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>

Signed-off-by: David Nind <david@davidnind.com>
Comment 22 David Nind 2021-05-20 21:13:46 UTC
Created attachment 121261 [details] [review]
Bug 15067: Add additional languages to advanced search language search

run update database script, look for new laguages under 'Language:' in Advanced search.
Languages that should be added are:
1. Estonian
2. Latvian
3. Lithuanian
4. Inuktitut
5. Inupiaq

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: David Nind <david@davidnind.com>
Comment 23 David Nind 2021-05-20 21:13:51 UTC
Created attachment 121262 [details] [review]
Bug 15067: Added translations of languages to those 5 languages

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.

Test Plan:
1. From homepage, go to Advanced search
2. Click on More options next to the Search button
3. Scroll down to the Limits section
4. Click on Language: No limit
5. Find Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq at the
bottom of the dropdown.
6. They should all be there.
7. Estonian and Latvian are repeated, this need to be fixed.
8. They need to be put in alphabetical order.

Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
The five languages are present, sorted alphabetically, and are not
repeated.

Signed-off-by: David Nind <david@davidnind.com>
Comment 24 Nick Clemens 2021-07-23 16:12:28 UTC
Created attachment 123140 [details] [review]
Bug 15067: Follow up to fix sorting

It looks like C4::Languages:getLanguages relies on implicit database
order for languages. They appear to be sorted by subtag so this patch
adds that sorting to the database query to ensure the dropdown is
sorted.

Signed-off-by: Aleisha <aleishaamohia@hotmail.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 25 Nick Clemens 2021-07-23 16:12:33 UTC
Created attachment 123141 [details] [review]
Bug 15067: Add additional languages to advanced search language search

run update database script, look for new laguages under 'Language:' in Advanced search.
Languages that should be added are:
1. Estonian
2. Latvian
3. Lithuanian
4. Inuktitut
5. Inupiaq

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 26 Nick Clemens 2021-07-23 16:12:37 UTC
Created attachment 123142 [details] [review]
Bug 15067: Added translations of languages to those 5 languages

Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq are added with
their translations to the Advanced Search languages.

Test Plan:
1. From homepage, go to Advanced search
2. Click on More options next to the Search button
3. Scroll down to the Limits section
4. Click on Language: No limit
5. Find Estonian, Latvian, Lithuanian, Inuktitut and Inupiaq at the
bottom of the dropdown.
6. They should all be there.
7. Estonian and Latvian are repeated, this need to be fixed.
8. They need to be put in alphabetical order.

Signed-off-by: Hayley Mapley <hayleymapley@catalyst.net.nz>
The five languages are present, sorted alphabetically, and are not
repeated.

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 27 Nick Clemens 2021-07-23 16:12:40 UTC
Created attachment 123143 [details] [review]
Bug 15067: (QA follow-up) Ensure update is idempotent

These tables really shoul;d have some unique keys, we need a test so we can add
those

This ended up being a bigger work than expected, RM feel free to reject or move
to another bug and let the risk of duplicated languages

This is an old one, let's get it in :-)

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 28 Jonathan Druart 2021-08-04 09:32:26 UTC
About the warns in the db rev, don't we actually want to explode instead?
Comment 29 Jonathan Druart 2021-08-04 12:05:39 UTC
Pushed to master for 21.11, thanks to everybody involved!
Comment 30 Jonathan Druart 2021-08-04 12:06:06 UTC
(In reply to Jonathan Druart from comment #28)
> About the warns in the db rev, don't we actually want to explode instead?

DB rev adjusted.
Comment 31 Tomás Cohen Arazi 2021-11-30 13:59:39 UTC
Created attachment 128105 [details] [review]
Bug 15067: (QA follow-up) Add missing UNIQUE constraint in kohastructure.sql

To test:
1. Run:
   $ reset_all
   $ koha-mysql kohadev
   > SHOW CREATE TABLE language_descriptions;
=> FAIL: There is KEY `subtag_type_lang` (`subtag`,`type`,`lang`)
2. Apply this patch
3. Repeat 1
=> SUCCESS: UNIQUE KEY `subtag_type_lang` (`subtag`,`type`,`lang`)

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 32 Tomás Cohen Arazi 2021-11-30 14:01:13 UTC
The latest patch should be pushed before the 21.11 packages are out. If not, a new DBREv is needed for fresh 21.11.00 installs. Upgrading works fine.
Comment 33 David Nind 2021-11-30 16:39:34 UTC
Created attachment 128112 [details] [review]
Bug 15067: (QA follow-up) Add missing UNIQUE constraint in kohastructure.sql

To test:
1. Run:
   $ reset_all
   $ koha-mysql kohadev
   > SHOW CREATE TABLE language_descriptions;
=> FAIL: There is KEY `subtag_type_lang` (`subtag`,`type`,`lang`)
2. Apply this patch
3. Repeat 1
=> SUCCESS: UNIQUE KEY `subtag_type_lang` (`subtag`,`type`,`lang`)

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>

Signed-off-by: David Nind <david@davidnind.com>
Comment 34 Tomás Cohen Arazi 2021-11-30 20:38:18 UTC
(In reply to Tomás Cohen Arazi from comment #32)
> The latest patch should be pushed before the 21.11 packages are out. If not,
> a new DBREv is needed for fresh 21.11.00 installs. Upgrading works fine.

We agreed the patch can wait for .01 so I'm moving to a separate bug, to better handle it by the QA team.
Comment 35 Manos PETRIDIS 2021-12-08 22:03:58 UTC
Has anyone noticed bugs 29631 and 29641? They seem to be caused by the unique keys introduced here. Shouldn't this bug revert to QA needed status?
Comment 36 MD NAZRUL ISLAM 2021-12-15 10:47:05 UTC
Upgrade to 21.06.00.012 [16:41:31]: Bug 15067 - Add missing languages
ERROR: {UNKNOWN}: DBI Exception: DBD::mysql::db do failed: Duplicate entry 'IN-region' for key 'uniq_lang' at /usr/share/koha/lib/C4/Installer.pm line 738
Comment 37 Jonathan Druart 2021-12-15 10:56:18 UTC
(In reply to MD NAZRUL ISLAM from comment #36)
> Upgrade to 21.06.00.012 [16:41:31]: Bug 15067 - Add missing languages
> ERROR: {UNKNOWN}: DBI Exception: DBD::mysql::db do failed: Duplicate entry
> 'IN-region' for key 'uniq_lang' at /usr/share/koha/lib/C4/Installer.pm line
> 738

See bug 29631.