Bug 26832 - Elasticsearch mappings export should use UTF-8
Summary: Elasticsearch mappings export should use UTF-8
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Fridolin Somers
QA Contact:
URL:
Keywords:
Depends on: 23204
Blocks:
  Show dependency treegraph
 
Reported: 2020-10-27 08:01 UTC by Fridolin Somers
Modified: 2020-11-17 20:53 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.06


Attachments
Bug 26832: Make Elasticsearch mappings export use UTF-8 (1.66 KB, patch)
2020-10-27 08:15 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 26832: Make Elasticsearch mappings export use UTF-8 (1.71 KB, patch)
2020-11-02 04:08 UTC, David Nind
Details | Diff | Splinter Review
Bug 26832: Make Elasticsearch mappings export use UTF-8 (1.77 KB, patch)
2020-11-03 08:56 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2020-10-27 08:01:49 UTC
Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file.
This export should use UTF-8 encoding, like other exports.
Comment 1 Fridolin Somers 2020-10-27 08:15:39 UTC
Created attachment 112546 [details] [review]
Bug 26832: Make Elasticsearch mappings export use UTF-8

Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file.
This export should use UTF-8 encoding, like other exports.

Test plan :
1) Go to Administration > Search engine configuration (Elasticsearch)
2) Edit a field label to use a diacrtic, for example local-number => Numéro
3) Save
4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings'
5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR
6) Reset memcached and plack
7) Back to Administration > Search engine configuration (Elasticsearch)
8) Click on 'Reset Mappings' and accept
9) Look at field 'local-number'
=> Without patch diacritic 'é' is broken
10) You may try with an emoji B-)
Comment 2 David Nind 2020-11-02 04:08:28 UTC
Created attachment 112813 [details] [review]
Bug 26832: Make Elasticsearch mappings export use UTF-8

Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file.
This export should use UTF-8 encoding, like other exports.

Test plan :
1) Go to Administration > Search engine configuration (Elasticsearch)
2) Edit a field label to use a diacrtic, for example local-number => Numéro
3) Save
4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings'
5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR
6) Reset memcached and plack
7) Back to Administration > Search engine configuration (Elasticsearch)
8) Click on 'Reset Mappings' and accept
9) Look at field 'local-number'
=> Without patch diacritic 'é' is broken
10) You may try with an emoji B-)

Signed-off-by: David Nind <david@davidnind.com>
Comment 3 David Nind 2020-11-02 04:14:45 UTC
I hope I tested this right!

I followed the test plan without applying the patch. I got an error with resetting the mappings at steps 8-9:

DBIx::Class::Storage::DBI::_dbh_execute(): Incorrect string value: '\xE9ro' for column 'label' at row 1 at /kohadevbox/koha/Koha/Objects.pm line 114
 at /usr/share/perl5/DBIx/Class/Exception.pm line 77

Notes for testing using koha-testing-docker:

1. Before starting testing setup elastic search as the search engine:
   i. Enable Elasticsearch: system preference SearchEngine
   ii. Reindex: koha-elasticsearch --rebuild -b -a kohadev
2. For step 4:
   i. Uncomment in /etc/koha/sites/kohadev/koha-conf.xml: <elasticsearch_index_mappings>
   ii. Note the location of the file: /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml
   iii. Create the directories: searchengine and elasticsearch
3. For step 5: 
   i. misc/search_tools/export_elasticsearch_mappings.pl -t marc21 > /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml
   ii. Check that the mappings.yaml file was created
   iii. Change ownership: chown -R kohadev-koha:kohadev-koha /etc/koha/sites/kohadev/searchengine
4. I got errors in the user interface, a trace error page when trying to Reset Mappings, and when trying to reindex.
5. I got past this by:
   i. applying the patch
   ii. disabling <elasticsearch_index_mappings> in koha-conf.xml
   iii. flush_memcached, restart_all 
   iv. Reset Mappings
   v. rerunning through the test plan (see notes 2 and 3 above)
6. To check if the file is UTF-8 use uchardet command:
   i. Install uchardet: apt-get install uchardet
   ii. Check the file format: uchardet /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml
   iii. It should show as being UTF-8
Comment 4 Martin Renvoize 2020-11-03 08:56:17 UTC
Created attachment 112899 [details] [review]
Bug 26832: Make Elasticsearch mappings export use UTF-8

Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file.
This export should use UTF-8 encoding, like other exports.

Test plan :
1) Go to Administration > Search engine configuration (Elasticsearch)
2) Edit a field label to use a diacrtic, for example local-number => Numéro
3) Save
4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings'
5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR
6) Reset memcached and plack
7) Back to Administration > Search engine configuration (Elasticsearch)
8) Click on 'Reset Mappings' and accept
9) Look at field 'local-number'
=> Without patch diacritic 'é' is broken
10) You may try with an emoji B-)

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 5 Martin Renvoize 2020-11-03 08:56:52 UTC
Passing QA
Comment 6 Jonathan Druart 2020-11-04 12:04:37 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 7 Lucas Gass 2020-11-16 21:07:51 UTC
backported to 20.05.x for 20.05.06
Comment 8 Aleisha Amohia 2020-11-17 05:27:06 UTC
missing dependencies, not backported to 19.11.x