Bug 25957

Summary: Elasticsearch 5.X - empty subfields cause error on suggestible fields
Product: Koha Reporter: Nick Clemens <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact:
Severity: normal    
Priority: P5 - low CC: aleisha, jonathan.druart, lucas, martin.renvoize
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.05
Attachments: Bug 25957: Unit test
Bug 25957: Don't push blank field values to ES document
Bug 25957: Unit test
Bug 25957: Don't push blank field values to ES document
Bug 25957: Unit test
Bug 25957: Don't push blank field values to ES document
Bug 25957: Unit test
Bug 25957: Don't push blank field values to ES document

Description Nick Clemens 2020-07-08 17:29:26 UTC
To recreate:
1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
2 - For example 'author' / 100a
    100 _ _ ‡a
3 - Index that record into Elasticsearch 5.X:
    perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
4 - Note error 'value must have length > 0'
5 - Edit mappings to set author 100a not suggestible
6 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
7 - Success
Comment 1 Nick Clemens 2020-07-08 17:31:40 UTC
Created attachment 106660 [details] [review]
Bug 25957: Unit test
Comment 2 Nick Clemens 2020-07-08 17:31:43 UTC
Created attachment 106661 [details] [review]
Bug 25957: Don't push blank field values to ES document

To test:
 1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
 2 - For example 'author' / 100a
     100 _ _ ‡a
 3 - Index that record into Elasticsearch 5.X:
     perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 4 - Note error 'value must have length > 0'
 5 - Edit mappings to set author 100a not suggestible
 6 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 7 - Success
 8 - Set field to suggestible again
 9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
11 - Success!
Comment 3 Nick Clemens 2020-09-28 15:31:05 UTC
Created attachment 110877 [details] [review]
Bug 25957: Unit test
Comment 4 Nick Clemens 2020-09-28 15:31:17 UTC
Created attachment 110878 [details] [review]
Bug 25957: Don't push blank field values to ES document

To test:
 1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
 2 - For example 'author' / 100a
     100 _ _ ‡a
 3 - Index that record into Elasticsearch 5.X:
     perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 4 - Note error 'value must have length > 0'
 5 - Edit mappings to set author 100a not suggestible
 6 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 7 - Success
 8 - Set field to suggestible again
 9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
11 - Success!
Comment 5 Andrew Fuerste-Henry 2020-09-28 16:41:32 UTC
Created attachment 110880 [details] [review]
Bug 25957: Unit test

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Comment 6 Andrew Fuerste-Henry 2020-09-28 16:41:36 UTC
Created attachment 110881 [details] [review]
Bug 25957: Don't push blank field values to ES document

To test:
 1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
 2 - For example 'author' / 100a
     100 _ _ ‡a
 3 - Index that record into Elasticsearch 5.X:
     perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 4 - Note error 'value must have length > 0'
 5 - Edit mappings to set author 100a not suggestible
 6 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 7 - Success
 8 - Set field to suggestible again
 9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
11 - Success!

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Comment 7 Martin Renvoize 2020-10-05 14:56:55 UTC
Created attachment 111240 [details] [review]
Bug 25957: Unit test

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 8 Martin Renvoize 2020-10-05 14:56:58 UTC
Created attachment 111241 [details] [review]
Bug 25957: Don't push blank field values to ES document

To test:
 1 - Load the sample DB or edit a record (using advanced cataloging editor) to have a blank subfield in a field that is indexed as suggestible
 2 - For example 'author' / 100a
     100 _ _ ‡a
 3 - Index that record into Elasticsearch 5.X:
     perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 4 - Note error 'value must have length > 0'
 5 - Edit mappings to set author 100a not suggestible
 6 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
 7 - Success
 8 - Set field to suggestible again
 9 - Apply patch
10 - perl misc/search_tools/rebuild_elasticsearch.pl  -v -bn 115 -b -d
11 - Success!

Signed-off-by: Bob Bennhoff <bbennhoff@clicweb.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize 2020-10-05 14:58:58 UTC
Works as expected, no regressions found.

Passing QA
Comment 10 Jonathan Druart 2020-10-08 09:37:52 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 11 Lucas Gass 2020-10-20 19:57:57 UTC
backported to 20.05.x for 20.05.05
Comment 12 Aleisha Amohia 2020-10-27 04:34:52 UTC
doesn't apply cleanly to 19.11.x, please rebase if needed