Bug 18318 - Wrong unicode tokenization
Summary: Wrong unicode tokenization
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Nick Clemens
QA Contact: Julian Maurice
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-03-23 08:48 UTC by Nick Clemens
Modified: 2019-06-27 09:24 UTC (History)
8 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 18318 - Unicode support for Elasticsearch (1.91 KB, patch)
2017-04-08 03:16 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 18318: Unicode support for Elasticsearch (1.97 KB, patch)
2017-09-25 13:10 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 18318: Unicode support for Elasticsearch (2.03 KB, patch)
2017-09-29 13:24 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2017-03-23 08:48:08 UTC
We need to offer unicode support in Elasticsearch

https://www.elastic.co/guide/en/elasticsearch/guide/current/case-folding.html
https://www.elastic.co/guide/en/elasticsearch/guide/current/icu-plugin.html

Currently a search for:
pokemon

Will not return results for:
pokémon


That is just an example :-)
Comment 1 Nick Clemens 2017-04-08 03:16:49 UTC
Created attachment 61982 [details] [review]
Bug 18318 - Unicode support for Elasticsearch

You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works
Comment 2 Tomás Cohen Arazi 2017-09-12 18:55:55 UTC
Just a note, the ICU analysis plugin has been incorporated into KohaDevBox so in order to test this patch you can provision a new box as usual:

KOHA_ELASTICSEARCH=1 vagrant up

Regards
Comment 3 Tomás Cohen Arazi 2017-09-25 13:10:17 UTC
Created attachment 67344 [details] [review]
Bug 18318: Unicode support for Elasticsearch

You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 4 Julian Maurice 2017-09-29 12:35:45 UTC
The patch does what it says, but it adds a dependency (ICU plugin) without mentioning it anywhere (AFAIK).
I don't know what is the best place for this (https://wiki.koha-community.org/wiki/Elasticsearch maybe ?), but it must be written somewhere.

Marking as Failed QA for now. I will pass QA as soon as there is some kind of documentation for this.
Comment 5 Tomás Cohen Arazi 2017-09-29 13:15:04 UTC
(In reply to Julian Maurice from comment #4)
> The patch does what it says, but it adds a dependency (ICU plugin) without
> mentioning it anywhere (AFAIK).
> I don't know what is the best place for this
> (https://wiki.koha-community.org/wiki/Elasticsearch maybe ?), but it must be
> written somewhere.
> 
> Marking as Failed QA for now. I will pass QA as soon as there is some kind
> of documentation for this.

The commit message mentions itbut definitely worth documenting in a more publicly available place.

I've adjusted the docs (https://wiki.koha-community.org/wiki/Elasticsearch).
Comment 6 Julian Maurice 2017-09-29 13:24:07 UTC
(In reply to Tomás Cohen Arazi from comment #5)
> I've adjusted the docs (https://wiki.koha-community.org/wiki/Elasticsearch).
Thanks!
Comment 7 Julian Maurice 2017-09-29 13:24:11 UTC
Created attachment 67449 [details] [review]
Bug 18318: Unicode support for Elasticsearch

You must install the icu plugin for elasticsearch
https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu.html

Once installed, apply this patch
Reindex your data, deleting the existing indexes
perl /home/vagrant/kohaclone/misc/search_tools/rebuild_elastic_search.pl
-d
Find (or add) some titles with accented characters
Verify that a search for the exact character or the unaccented version
works

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Signed-off-by: Julian Maurice <julian.maurice@biblibre.com>
Comment 8 Jonathan Druart 2017-09-29 15:45:54 UTC
Pushed to master for 17.11, thanks to everybody involved!
Comment 9 Fridolin Somers 2017-10-17 07:42:07 UTC
ICU support, awesome :D, thanks all.
Comment 10 Fridolin Somers 2017-10-17 07:42:31 UTC
Pushed to 17.05.x, will be in 17.05.05.
Comment 11 Katrin Fischer 2017-10-18 21:14:00 UTC
Patch does not apply cleanly. Closing as 16.11.x is missing quite some patches for Elasticsearch now.