If a record contains malformed data that is supposed to be indexed in an integer field, the document is rejected by elasticsearch. Elasticsearch should ignore malformed data.
Created attachment 106276 [details] [review] Bug 25873: Ignore malformed data for Elasticsearch integer fields If we try to put malformed data into an integer field, Elasticsearch rejects the whole document. Setting 'ignore_malformed' to true allows to ignore malformed data and process the other fields of the document normally https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html Test plan: * Without the patch 1. In search engine configuration, change the type of a text field to 'Number' (for instance 'title') 2. misc/search_tools/rebuild_elasticsearch.pl -d -b 3. See that the index is empty (unless you have titles consisting only of digits) * With the patch 1. misc/search_tools/rebuild_elasticsearch.pl -d -b 2. Now records are correctly indexed
Created attachment 106703 [details] [review] Bug 25873: Ignore malformed data for Elasticsearch integer fields If we try to put malformed data into an integer field, Elasticsearch rejects the whole document. Setting 'ignore_malformed' to true allows to ignore malformed data and process the other fields of the document normally https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html Test plan: * Without the patch 1. In search engine configuration, change the type of a text field to 'Number' (for instance 'title') 2. misc/search_tools/rebuild_elasticsearch.pl -d -b 3. See that the index is empty (unless you have titles consisting only of digits) * With the patch 1. misc/search_tools/rebuild_elasticsearch.pl -d -b 2. Now records are correctly indexed Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Trivial config change that can be overridden if desired, passing QA
Pushed to master for 20.11, thanks to everybody involved!
backported to 20.05.x for 20.05.03
backported to 19.11.x for 19.11.09
Backported to 19.05.x branch for 19.05.15