Bug 18318

Summary: Wrong unicode tokenization
Product: Koha Reporter: Nick Clemens <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Julian Maurice <julian.maurice>
Severity: major    
Priority: P5 - low CC: claire_gravely, fridolin.somers, jonathan.druart, julian.maurice, katrin.fischer, philippe.blouin, severine.queune, tomascohen
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Attachments: Bug 18318 - Unicode support for Elasticsearch
Bug 18318: Unicode support for Elasticsearch
Bug 18318: Unicode support for Elasticsearch

Description Nick Clemens 2017-03-23 08:48:08 UTC
We need to offer unicode support in Elasticsearch

https://www.elastic.co/guide/en/elasticsearch/guide/current/case-folding.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-plugin.html

Currently a search for:
pokemon

Will not return results for:
pokémon


That is just an example :-)
Comment 1 Nick Clemens 2017-04-08 03:16:49 UTC
Created attachment 61982 [details] [review]
Bug 18318 - Unicode support for Elasticsearch

You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works
Comment 2 Tomás Cohen Arazi 2017-09-12 18:55:55 UTC
Just a note, the ICU analysis plugin has been incorporated into KohaDevBox so in order to test this patch you can provision a new box as usual:

KOHA_ELASTICSEARCH=1 vagrant up

Regards
Comment 3 Tomás Cohen Arazi 2017-09-25 13:10:17 UTC
Created attachment 67344 [details] [review]
Bug 18318: Unicode support for Elasticsearch

You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 4 Julian Maurice 2017-09-29 12:35:45 UTC
The patch does what it says, but it adds a dependency (ICU plugin) without mentioning it anywhere (AFAIK).
I don't know what is the best place for this (https://wiki.koha-community.org/wiki/Elasticsearch maybe ?), but it must be written somewhere.

Marking as Failed QA for now. I will pass QA as soon as there is some kind of documentation for this.
Comment 5 Tomás Cohen Arazi 2017-09-29 13:15:04 UTC
(In reply to Julian Maurice from comment #4)
> The patch does what it says, but it adds a dependency (ICU plugin) without
> mentioning it anywhere (AFAIK).
> I don't know what is the best place for this
> (https://wiki.koha-community.org/wiki/Elasticsearch maybe ?), but it must be
> written somewhere.
> 
> Marking as Failed QA for now. I will pass QA as soon as there is some kind
> of documentation for this.

The commit message mentions itbut definitely worth documenting in a more publicly available place.

I've adjusted the docs (https://wiki.koha-community.org/wiki/Elasticsearch).
Comment 6 Julian Maurice 2017-09-29 13:24:07 UTC
(In reply to Tomás Cohen Arazi from comment #5)
> I've adjusted the docs (https://wiki.koha-community.org/wiki/Elasticsearch).
Thanks!
Comment 7 Julian Maurice 2017-09-29 13:24:11 UTC
Created attachment 67449 [details] [review]
Bug 18318: Unicode support for Elasticsearch

You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 8 Jonathan Druart 2017-09-29 15:45:54 UTC
Pushed to master for 17.11, thanks to everybody involved!
Comment 9 Fridolin Somers 2017-10-17 07:42:07 UTC
ICU support, awesome :D, thanks all.
Comment 10 Fridolin Somers 2017-10-17 07:42:31 UTC
Pushed to 17.05.x, will be in 17.05.05.
Comment 11 Katrin Fischer 2017-10-18 21:14:00 UTC
Patch does not apply cleanly. Closing as 16.11.x is missing quite some patches for Elasticsearch now.