Bug 25873 - Elasticsearch - Records with malformed data are not indexed
Summary: Elasticsearch - Records with malformed data are not indexed
Status: Pushed to oldoldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Julian Maurice
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-06-25 07:41 UTC by Julian Maurice
Modified: 2020-09-20 11:12 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.03, 19.11.09, 19.05.15


Attachments
Bug 25873: Ignore malformed data for Elasticsearch integer fields (1.43 KB, patch)
2020-06-25 07:42 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 25873: Ignore malformed data for Elasticsearch integer fields (1.49 KB, patch)
2020-07-09 11:20 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2020-06-25 07:41:25 UTC
If a record contains malformed data that is supposed to be indexed in an integer field, the document is rejected by elasticsearch.
Elasticsearch should ignore malformed data.
Comment 1 Julian Maurice 2020-06-25 07:42:22 UTC
Created attachment 106276 [details] [review]
Bug 25873: Ignore malformed data for Elasticsearch integer fields

If we try to put malformed data into an integer field, Elasticsearch
rejects the whole document.
Setting 'ignore_malformed' to true allows to ignore malformed data and
process the other fields of the document normally

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html

Test plan:
* Without the patch
  1. In search engine configuration, change the type of a text field to
  'Number' (for instance 'title')
  2. misc/search_tools/rebuild_elasticsearch.pl -d -b
  3. See that the index is empty (unless you have titles consisting only
  of digits)
* With the patch
  1. misc/search_tools/rebuild_elasticsearch.pl -d -b
  2. Now records are correctly indexed
Comment 2 Nick Clemens 2020-07-09 11:20:23 UTC
Created attachment 106703 [details] [review]
Bug 25873: Ignore malformed data for Elasticsearch integer fields

If we try to put malformed data into an integer field, Elasticsearch
rejects the whole document.
Setting 'ignore_malformed' to true allows to ignore malformed data and
process the other fields of the document normally

https://www.elastic.co/guide/en/elasticsearch/reference/7.8/ignore-malformed.html

Test plan:
* Without the patch
  1. In search engine configuration, change the type of a text field to
  'Number' (for instance 'title')
  2. misc/search_tools/rebuild_elasticsearch.pl -d -b
  3. See that the index is empty (unless you have titles consisting only
  of digits)
* With the patch
  1. misc/search_tools/rebuild_elasticsearch.pl -d -b
  2. Now records are correctly indexed

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 3 Nick Clemens 2020-07-31 10:45:55 UTC
Trivial config change that can be overridden if desired, passing QA
Comment 4 Jonathan Druart 2020-08-03 09:29:17 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 5 Lucas Gass 2020-08-13 17:06:22 UTC
backported to 20.05.x for 20.05.03
Comment 6 Aleisha Amohia 2020-08-16 21:01:44 UTC
backported to 19.11.x for 19.11.09
Comment 7 Victor Grousset/tuxayo 2020-09-20 11:11:58 UTC
Backported to 19.05.x branch for 19.05.15