Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports.
Created attachment 112546 [details] [review] Bug 26832: Make Elasticsearch mappings export use UTF-8 Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. Test plan : 1) Go to Administration > Search engine configuration (Elasticsearch) 2) Edit a field label to use a diacrtic, for example local-number => Numéro 3) Save 4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings' 5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR 6) Reset memcached and plack 7) Back to Administration > Search engine configuration (Elasticsearch) 8) Click on 'Reset Mappings' and accept 9) Look at field 'local-number' => Without patch diacritic 'é' is broken 10) You may try with an emoji B-)
Created attachment 112813 [details] [review] Bug 26832: Make Elasticsearch mappings export use UTF-8 Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. Test plan : 1) Go to Administration > Search engine configuration (Elasticsearch) 2) Edit a field label to use a diacrtic, for example local-number => Numéro 3) Save 4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings' 5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR 6) Reset memcached and plack 7) Back to Administration > Search engine configuration (Elasticsearch) 8) Click on 'Reset Mappings' and accept 9) Look at field 'local-number' => Without patch diacritic 'é' is broken 10) You may try with an emoji B-) Signed-off-by: David Nind <david@davidnind.com>
I hope I tested this right! I followed the test plan without applying the patch. I got an error with resetting the mappings at steps 8-9: DBIx::Class::Storage::DBI::_dbh_execute(): Incorrect string value: '\xE9ro' for column 'label' at row 1 at /kohadevbox/koha/Koha/Objects.pm line 114 at /usr/share/perl5/DBIx/Class/Exception.pm line 77 Notes for testing using koha-testing-docker: 1. Before starting testing setup elastic search as the search engine: i. Enable Elasticsearch: system preference SearchEngine ii. Reindex: koha-elasticsearch --rebuild -b -a kohadev 2. For step 4: i. Uncomment in /etc/koha/sites/kohadev/koha-conf.xml: <elasticsearch_index_mappings> ii. Note the location of the file: /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml iii. Create the directories: searchengine and elasticsearch 3. For step 5: i. misc/search_tools/export_elasticsearch_mappings.pl -t marc21 > /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml ii. Check that the mappings.yaml file was created iii. Change ownership: chown -R kohadev-koha:kohadev-koha /etc/koha/sites/kohadev/searchengine 4. I got errors in the user interface, a trace error page when trying to Reset Mappings, and when trying to reindex. 5. I got past this by: i. applying the patch ii. disabling <elasticsearch_index_mappings> in koha-conf.xml iii. flush_memcached, restart_all iv. Reset Mappings v. rerunning through the test plan (see notes 2 and 3 above) 6. To check if the file is UTF-8 use uchardet command: i. Install uchardet: apt-get install uchardet ii. Check the file format: uchardet /etc/koha/sites/kohadev/searchengine/elasticsearch/mappings.yaml iii. It should show as being UTF-8
Created attachment 112899 [details] [review] Bug 26832: Make Elasticsearch mappings export use UTF-8 Script misc/search_tools/export_elasticsearch_mappings.pl allows to export current search engine configuration into a YAML file. This export should use UTF-8 encoding, like other exports. Test plan : 1) Go to Administration > Search engine configuration (Elasticsearch) 2) Edit a field label to use a diacrtic, for example local-number => Numéro 3) Save 4) Edit file etc/koha-conf.xml to enable 'elasticsearch_index_mappings' 5) Export mappings to file via misc/search_tools/export_elasticsearch_mappings.pl -t $MARCFLAVOUR 6) Reset memcached and plack 7) Back to Administration > Search engine configuration (Elasticsearch) 8) Click on 'Reset Mappings' and accept 9) Look at field 'local-number' => Without patch diacritic 'é' is broken 10) You may try with an emoji B-) Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Passing QA
Pushed to master for 20.11, thanks to everybody involved!
backported to 20.05.x for 20.05.06
missing dependencies, not backported to 19.11.x