Summary: | Elasticsearch - Records with malformed data are not indexed | ||
---|---|---|---|
Product: | Koha | Reporter: | Julian Maurice <julian.maurice> |
Component: | Searching - Elasticsearch | Assignee: | Julian Maurice <julian.maurice> |
Status: | CLOSED FIXED | QA Contact: | |
Severity: | normal | ||
Priority: | P5 - low | CC: | aleisha, lucas, nick, victor |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
GIT URL: | Change sponsored?: | --- | |
Patch complexity: | --- | Documentation contact: | |
Documentation submission: | Text to go in the release notes: | ||
Version(s) released in: |
20.11.00, 20.05.03, 19.11.09, 19.05.15
|
Circulation function: | |
Attachments: |
Bug 25873: Ignore malformed data for Elasticsearch integer fields
Bug 25873: Ignore malformed data for Elasticsearch integer fields |
Description
Julian Maurice
2020-06-25 07:41:25 UTC
Created attachment 106276 [details] [review] Bug 25873: Ignore malformed data for Elasticsearch integer fields If we try to put malformed data into an integer field, Elasticsearch rejects the whole document. Setting 'ignore_malformed' to true allows to ignore malformed data and process the other fields of the document normally https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html Test plan: * Without the patch 1. In search engine configuration, change the type of a text field to 'Number' (for instance 'title') 2. misc/search_tools/rebuild_elasticsearch.pl -d -b 3. See that the index is empty (unless you have titles consisting only of digits) * With the patch 1. misc/search_tools/rebuild_elasticsearch.pl -d -b 2. Now records are correctly indexed Created attachment 106703 [details] [review] Bug 25873: Ignore malformed data for Elasticsearch integer fields If we try to put malformed data into an integer field, Elasticsearch rejects the whole document. Setting 'ignore_malformed' to true allows to ignore malformed data and process the other fields of the document normally https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html Test plan: * Without the patch 1. In search engine configuration, change the type of a text field to 'Number' (for instance 'title') 2. misc/search_tools/rebuild_elasticsearch.pl -d -b 3. See that the index is empty (unless you have titles consisting only of digits) * With the patch 1. misc/search_tools/rebuild_elasticsearch.pl -d -b 2. Now records are correctly indexed Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Trivial config change that can be overridden if desired, passing QA Pushed to master for 20.11, thanks to everybody involved! backported to 20.05.x for 20.05.03 backported to 19.11.x for 19.11.09 Backported to 19.05.x branch for 19.05.15 |