Bug 18213

Summary: Add language facets to Elasticsearch
Product: Koha Reporter: Nick Clemens <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Josef Moravec <josef.moravec>
Severity: enhancement    
Priority: P5 - low CC: alex.arnaud, david.bourgault, f.demians, jonathan.druart, josef.moravec, katrin.fischer, marjorie.barry-vila, martin.renvoize, nicolas.legrand, severine.queune, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18235
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=36437
Change sponsored?: Sponsored Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on: 18235    
Bug Blocks: 22449    
Attachments: Bug 18213 - Add language facets to Elasticsearch
Bug 18213 - DB Update (facet order)
Bug 18213 - Add Template plugin and fix for C4/Languages
Bug 18213 - Add language facets to Elasticsearch
Bug 18213 - DB Update (facet order)
Bug 18213 - Add Template plugin and fix for C4/Languages
Bug 18213 - Add language facets to Elasticsearch
Bug 18213 - DB Update (facet order)
Bug 18213 - Add Template plugin and fix for C4/Languages
Bug 18213: (follow-up) Correctly sort facets
Bug 18213 - Add language facets to Elasticsearch
Bug 18213 - DB Update (facet order)
Bug 18213 - Add Template plugin and fix for C4/Languages
Bug 18213: (follow-up) Correctly sort facets
Bug 18213 - Add language facets to Elasticsearch
Bug 18213 - DB Update (facet order)
Bug 18213 - Add Template plugin and fix for C4/Languages
Bug 18213: (follow-up) Correctly sort facets
Bug 18213: Add language facets to Elasticsearch
Bug 18213: DB Update (facet order)
Bug 18213: Add Template plugin and fix for C4/Languages
Bug 18213: (follow-up) Correctly sort facets

Description Nick Clemens 2017-03-05 16:00:11 UTC

    
Comment 1 Jonathan Druart 2017-03-09 14:14:25 UTC
Would be good to make it on top of bug 18235.
Comment 2 Nick Clemens 2017-07-05 12:16:41 UTC
Created attachment 64798 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Comment 3 Nick Clemens 2017-07-06 09:42:27 UTC
Created attachment 64841 [details] [review]
Bug 18213 - DB Update (facet order)
Comment 4 Nick Clemens 2017-07-07 10:46:08 UTC
Created attachment 64905 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages
Comment 5 David Bourgault 2017-11-03 20:38:33 UTC
Does not apply.
fatal: sha1 information is lacking or useless (Koha/SearchEngine/Elasticsearch.pm).
Comment 6 Nick Clemens 2017-11-04 02:10:42 UTC
Created attachment 68944 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Comment 7 Nick Clemens 2017-11-04 02:10:45 UTC
Created attachment 68945 [details] [review]
Bug 18213 - DB Update (facet order)
Comment 8 Nick Clemens 2017-11-04 02:10:48 UTC
Created attachment 68946 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages
Comment 9 Nicolas Legrand 2018-03-14 08:37:33 UTC
I don't dare to rebase it myself ^^
Comment 10 Nick Clemens 2018-03-15 12:54:49 UTC
Created attachment 72940 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Comment 11 Nick Clemens 2018-03-15 12:54:53 UTC
Created attachment 72941 [details] [review]
Bug 18213 - DB Update (facet order)
Comment 12 Nick Clemens 2018-03-15 12:54:57 UTC
Created attachment 72942 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages
Comment 13 Nick Clemens 2018-03-15 12:55:01 UTC
Created attachment 72943 [details] [review]
Bug 18213: (follow-up) Correctly sort facets

Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically
Comment 14 Nicolas Legrand 2018-03-16 10:06:35 UTC Comment hidden (obsolete)
Comment 15 Nicolas Legrand 2018-03-16 10:06:38 UTC Comment hidden (obsolete)
Comment 16 Nicolas Legrand 2018-03-16 10:06:41 UTC Comment hidden (obsolete)
Comment 17 Nicolas Legrand 2018-03-16 10:06:44 UTC Comment hidden (obsolete)
Comment 18 Alex Arnaud 2018-06-12 14:10:32 UTC
Created attachment 75971 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 19 Alex Arnaud 2018-06-12 14:10:40 UTC
Created attachment 75972 [details] [review]
Bug 18213 - DB Update (facet order)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 20 Alex Arnaud 2018-06-12 14:10:47 UTC
Created attachment 75973 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 21 Alex Arnaud 2018-06-12 14:10:55 UTC
Created attachment 75974 [details] [review]
Bug 18213: (follow-up) Correctly sort facets

Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 22 Josef Moravec 2019-03-04 14:01:14 UTC
dependency bug 18235 passed qa, setting this back to signed off
Comment 23 Josef Moravec 2019-03-04 14:48:42 UTC
Created attachment 85998 [details] [review]
Bug 18213: Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 24 Josef Moravec 2019-03-04 14:48:46 UTC
Created attachment 85999 [details] [review]
Bug 18213: DB Update (facet order)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 25 Josef Moravec 2019-03-04 14:48:50 UTC
Created attachment 86000 [details] [review]
Bug 18213: Add Template plugin and fix for C4/Languages

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 26 Josef Moravec 2019-03-04 14:48:54 UTC
Created attachment 86001 [details] [review]
Bug 18213: (follow-up) Correctly sort facets

Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 27 Nick Clemens 2019-03-28 16:01:30 UTC
Awesome work all!

Pushed to master for 19.05
Comment 28 Martin Renvoize 2019-04-01 13:57:42 UTC
Enhancement, will not be backported to 18.11.x series.
Comment 29 Frédéric Demians 2020-02-19 06:45:59 UTC
There is somehow a regression with ES language facets compared to Zebra facets. Zebra language facet is decoded from LANG authorized values list. ES language facet (thanks to this patch) is decoded from language_* tables. But there are more languages defined in LANG authorized values list: all old languages for example, like grc = Ancient Greek (to 1453).

Solution : add missing languages to language_* tables. But is it the purpose of those tables which are designed to allow Koha interface translation rather than storing MARC biblio records usable languages?
Comment 30 Katrin Fischer 2020-02-19 08:26:02 UTC
(In reply to Frédéric Demians from comment #29)
> There is somehow a regression with ES language facets compared to Zebra
> facets. Zebra language facet is decoded from LANG authorized values list. ES
> language facet (thanks to this patch) is decoded from language_* tables. But
> there are more languages defined in LANG authorized values list: all old
> languages for example, like grc = Ancient Greek (to 1453).
> 
> Solution : add missing languages to language_* tables. But is it the purpose
> of those tables which are designed to allow Koha interface translation
> rather than storing MARC biblio records usable languages?

Afaik they are used to build the list of languages on advanced search and also the language options when you have different langaugaes installed. The big win of the tables currently is that they are transalatable/offer the language name in the language of the language (hope that makes sense).
Comment 31 Katrin Fischer 2020-02-19 08:28:57 UTC
That said - are you sure the langauge facet is in standard Koha for Zebra? I don't see a definition for it in getfacets.