Bug 25873

Summary: Elasticsearch - Records with malformed data are not indexed
Product: Koha Reporter: Julian Maurice <julian.maurice>
Component: Searching - ElasticsearchAssignee: Julian Maurice <julian.maurice>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P5 - low CC: aleisha, lucas, nick, victor
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.03, 19.11.09, 19.05.15
Attachments: Bug 25873: Ignore malformed data for Elasticsearch integer fields
Bug 25873: Ignore malformed data for Elasticsearch integer fields

Description Julian Maurice 2020-06-25 07:41:25 UTC
If a record contains malformed data that is supposed to be indexed in an integer field, the document is rejected by elasticsearch.
Elasticsearch should ignore malformed data.
Comment 1 Julian Maurice 2020-06-25 07:42:22 UTC
Created attachment 106276 [details] [review]
Bug 25873: Ignore malformed data for Elasticsearch integer fields

If we try to put malformed data into an integer field, Elasticsearch
rejects the whole document.
Setting 'ignore_malformed' to true allows to ignore malformed data and
process the other fields of the document normally

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html

Test plan:
* Without the patch
  1. In search engine configuration, change the type of a text field to
  'Number' (for instance 'title')
  2. misc/search_tools/rebuild_elasticsearch.pl -d -b
  3. See that the index is empty (unless you have titles consisting only
  of digits)
* With the patch
  1. misc/search_tools/rebuild_elasticsearch.pl -d -b
  2. Now records are correctly indexed
Comment 2 Nick Clemens 2020-07-09 11:20:23 UTC
Created attachment 106703 [details] [review]
Bug 25873: Ignore malformed data for Elasticsearch integer fields

If we try to put malformed data into an integer field, Elasticsearch
rejects the whole document.
Setting 'ignore_malformed' to true allows to ignore malformed data and
process the other fields of the document normally

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html

Test plan:
* Without the patch
  1. In search engine configuration, change the type of a text field to
  'Number' (for instance 'title')
  2. misc/search_tools/rebuild_elasticsearch.pl -d -b
  3. See that the index is empty (unless you have titles consisting only
  of digits)
* With the patch
  1. misc/search_tools/rebuild_elasticsearch.pl -d -b
  2. Now records are correctly indexed

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 3 Nick Clemens 2020-07-31 10:45:55 UTC
Trivial config change that can be overridden if desired, passing QA
Comment 4 Jonathan Druart 2020-08-03 09:29:17 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 5 Lucas Gass 2020-08-13 17:06:22 UTC
backported to 20.05.x for 20.05.03
Comment 6 Aleisha Amohia 2020-08-16 21:01:44 UTC
backported to 19.11.x for 19.11.09
Comment 7 Victor Grousset/tuxayo 2020-09-20 11:11:58 UTC
Backported to 19.05.x branch for 19.05.15