Bug 18213 - Add language facets to Elasticsearch
Summary: Add language facets to Elasticsearch
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Nick Clemens
QA Contact: Josef Moravec
URL:
Keywords: release-notes-needed
Depends on: 18235
Blocks: 22449
  Show dependency treegraph
 
Reported: 2017-03-05 16:00 UTC by Nick Clemens
Modified: 2020-02-19 08:28 UTC (History)
11 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 18213 - Add language facets to Elasticsearch (10.47 KB, patch)
2017-07-05 12:16 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - DB Update (facet order) (916 bytes, patch)
2017-07-06 09:42 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - Add Template plugin and fix for C4/Languages (2.84 KB, patch)
2017-07-07 10:46 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - Add language facets to Elasticsearch (14.08 KB, patch)
2017-11-04 02:10 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - DB Update (facet order) (916 bytes, patch)
2017-11-04 02:10 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - Add Template plugin and fix for C4/Languages (2.84 KB, patch)
2017-11-04 02:10 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - Add language facets to Elasticsearch (10.66 KB, patch)
2018-03-15 12:54 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - DB Update (facet order) (916 bytes, patch)
2018-03-15 12:54 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - Add Template plugin and fix for C4/Languages (2.84 KB, patch)
2018-03-15 12:54 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213: (follow-up) Correctly sort facets (925 bytes, patch)
2018-03-15 12:55 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18213 - Add language facets to Elasticsearch (10.52 KB, patch)
2018-03-16 10:06 UTC, Nicolas Legrand
Details | Diff | Splinter Review
Bug 18213 - DB Update (facet order) (974 bytes, patch)
2018-03-16 10:06 UTC, Nicolas Legrand
Details | Diff | Splinter Review
Bug 18213 - Add Template plugin and fix for C4/Languages (2.90 KB, patch)
2018-03-16 10:06 UTC, Nicolas Legrand
Details | Diff | Splinter Review
Bug 18213: (follow-up) Correctly sort facets (984 bytes, patch)
2018-03-16 10:06 UTC, Nicolas Legrand
Details | Diff | Splinter Review
Bug 18213 - Add language facets to Elasticsearch (10.57 KB, patch)
2018-06-12 14:10 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 18213 - DB Update (facet order) (1.00 KB, patch)
2018-06-12 14:10 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 18213 - Add Template plugin and fix for C4/Languages (2.95 KB, patch)
2018-06-12 14:10 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 18213: (follow-up) Correctly sort facets (1.01 KB, patch)
2018-06-12 14:10 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 18213: Add language facets to Elasticsearch (10.33 KB, patch)
2019-03-04 14:48 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 18213: DB Update (facet order) (1.06 KB, patch)
2019-03-04 14:48 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 18213: Add Template plugin and fix for C4/Languages (3.02 KB, patch)
2019-03-04 14:48 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 18213: (follow-up) Correctly sort facets (1.08 KB, patch)
2019-03-04 14:48 UTC, Josef Moravec
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2017-03-05 16:00:11 UTC

    
Comment 1 Jonathan Druart 2017-03-09 14:14:25 UTC
Would be good to make it on top of bug 18235.
Comment 2 Nick Clemens 2017-07-05 12:16:41 UTC
Created attachment 64798 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Comment 3 Nick Clemens 2017-07-06 09:42:27 UTC
Created attachment 64841 [details] [review]
Bug 18213 - DB Update (facet order)
Comment 4 Nick Clemens 2017-07-07 10:46:08 UTC
Created attachment 64905 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages
Comment 5 David Bourgault 2017-11-03 20:38:33 UTC
Does not apply.
fatal: sha1 information is lacking or useless (Koha/SearchEngine/Elasticsearch.pm).
Comment 6 Nick Clemens 2017-11-04 02:10:42 UTC
Created attachment 68944 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Comment 7 Nick Clemens 2017-11-04 02:10:45 UTC
Created attachment 68945 [details] [review]
Bug 18213 - DB Update (facet order)
Comment 8 Nick Clemens 2017-11-04 02:10:48 UTC
Created attachment 68946 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages
Comment 9 Nicolas Legrand 2018-03-14 08:37:33 UTC
I don't dare to rebase it myself ^^
Comment 10 Nick Clemens 2018-03-15 12:54:49 UTC
Created attachment 72940 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t
Comment 11 Nick Clemens 2018-03-15 12:54:53 UTC
Created attachment 72941 [details] [review]
Bug 18213 - DB Update (facet order)
Comment 12 Nick Clemens 2018-03-15 12:54:57 UTC
Created attachment 72942 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages
Comment 13 Nick Clemens 2018-03-15 12:55:01 UTC
Created attachment 72943 [details] [review]
Bug 18213: (follow-up) Correctly sort facets

Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically
Comment 14 Nicolas Legrand 2018-03-16 10:06:35 UTC Comment hidden (obsolete)
Comment 15 Nicolas Legrand 2018-03-16 10:06:38 UTC Comment hidden (obsolete)
Comment 16 Nicolas Legrand 2018-03-16 10:06:41 UTC Comment hidden (obsolete)
Comment 17 Nicolas Legrand 2018-03-16 10:06:44 UTC Comment hidden (obsolete)
Comment 18 Alex Arnaud 2018-06-12 14:10:32 UTC
Created attachment 75971 [details] [review]
Bug 18213 - Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 19 Alex Arnaud 2018-06-12 14:10:40 UTC
Created attachment 75972 [details] [review]
Bug 18213 - DB Update (facet order)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 20 Alex Arnaud 2018-06-12 14:10:47 UTC
Created attachment 75973 [details] [review]
Bug 18213 - Add Template plugin and fix for C4/Languages

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 21 Alex Arnaud 2018-06-12 14:10:55 UTC
Created attachment 75974 [details] [review]
Bug 18213: (follow-up) Correctly sort facets

Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 22 Josef Moravec 2019-03-04 14:01:14 UTC
dependency bug 18235 passed qa, setting this back to signed off
Comment 23 Josef Moravec 2019-03-04 14:48:42 UTC
Created attachment 85998 [details] [review]
Bug 18213: Add language facets to Elasticsearch

This patch add language as a facet to ES results - it adds
a new template plugin for languages to get the appropriate
description given an iso 639-2 code

To test:
1 - Make sure you have records with differing languages (in the MARC21 008
        field characters 35-37 or UNIMARC 101a)
2 - Apply patch
3 - Reload Elasticsearch settings:
http://localhost:8081/cgi-bin/koha/admin/searchengine/elasticsearch/mappings.pl?op=reset&i_know_what_i_am_doing=1
4 - Reindex your records
5 - Search for a phrase that will return results in several languages
6 - Verify you see factes correctly labelled for 'Language'
7 - Verify the facets work
8 - Verify both opac and staff results
9 - prove t/db_dependent/Languages.t

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 24 Josef Moravec 2019-03-04 14:48:46 UTC
Created attachment 85999 [details] [review]
Bug 18213: DB Update (facet order)

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 25 Josef Moravec 2019-03-04 14:48:50 UTC
Created attachment 86000 [details] [review]
Bug 18213: Add Template plugin and fix for C4/Languages

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 26 Josef Moravec 2019-03-04 14:48:54 UTC
Created attachment 86001 [details] [review]
Bug 18213: (follow-up) Correctly sort facets

Previously we sorted alphabetically - that works for 1-9.
With a 10th facet category we should sort numerically

Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 27 Nick Clemens 2019-03-28 16:01:30 UTC
Awesome work all!

Pushed to master for 19.05
Comment 28 Martin Renvoize 2019-04-01 13:57:42 UTC
Enhancement, will not be backported to 18.11.x series.
Comment 29 Frédéric Demians 2020-02-19 06:45:59 UTC
There is somehow a regression with ES language facets compared to Zebra facets. Zebra language facet is decoded from LANG authorized values list. ES language facet (thanks to this patch) is decoded from language_* tables. But there are more languages defined in LANG authorized values list: all old languages for example, like grc = Ancient Greek (to 1453).

Solution : add missing languages to language_* tables. But is it the purpose of those tables which are designed to allow Koha interface translation rather than storing MARC biblio records usable languages?
Comment 30 Katrin Fischer 2020-02-19 08:26:02 UTC
(In reply to Frédéric Demians from comment #29)
> There is somehow a regression with ES language facets compared to Zebra
> facets. Zebra language facet is decoded from LANG authorized values list. ES
> language facet (thanks to this patch) is decoded from language_* tables. But
> there are more languages defined in LANG authorized values list: all old
> languages for example, like grc = Ancient Greek (to 1453).
> 
> Solution : add missing languages to language_* tables. But is it the purpose
> of those tables which are designed to allow Koha interface translation
> rather than storing MARC biblio records usable languages?

Afaik they are used to build the list of languages on advanced search and also the language options when you have different langaugaes installed. The big win of the tables currently is that they are transalatable/offer the language name in the language of the language (hope that makes sense).
Comment 31 Katrin Fischer 2020-02-19 08:28:57 UTC
That said - are you sure the langauge facet is in standard Koha for Zebra? I don't see a definition for it in getfacets.