Bug 22246 - Elasticsearch indexing needs a maximum length for `__sort` fields
Summary: Elasticsearch indexing needs a maximum length for `__sort` fields
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P3 normal (vote)
Assignee: Ere Maijala
QA Contact: Josef Moravec
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-01-31 12:22 UTC by Ere Maijala
Modified: 2019-10-14 19:59 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 22246: Fix indexing of large fields with Elasticsearch (11.03 KB, patch)
2019-02-01 11:25 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 22246: Fix indexing of large fields with Elasticsearch (10.62 KB, patch)
2019-02-01 11:38 UTC, Ere Maijala
Details | Diff | Splinter Review
Bug 22246: Fix indexing of large fields with Elasticsearch (10.68 KB, patch)
2019-02-01 13:54 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22246: Fix indexing of large fields with Elasticsearch (10.74 KB, patch)
2019-02-28 13:13 UTC, Josef Moravec
Details | Diff | Splinter Review
Bug 22246: Fix indexing of large fields with Elasticsearch (11.58 KB, patch)
2019-03-05 09:13 UTC, Ere Maijala
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Ere Maijala 2019-01-31 12:22:42 UTC
Currently there's no limit to the length of *__sort fields. If such field is created from item information or something else that can be repeated a lot, the field may become too long. Such field is e.g. local-classification, which is by default created from 952o. If there are hundreds or thousands of items, the field may become longer than what Elasticsearch accepts. Also consider making it a non-sortable field by default.
Comment 1 Ere Maijala 2019-02-01 11:25:16 UTC
Created attachment 84607 [details] [review]
Bug 22246: Fix indexing of large fields with Elasticsearch

Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Also fixes mapping for llevel while at it.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Comment 2 Ere Maijala 2019-02-01 11:38:48 UTC
Created attachment 84608 [details] [review]
Bug 22246: Fix indexing of large fields with Elasticsearch

Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Comment 3 Nick Clemens 2019-02-01 13:54:05 UTC
Created attachment 84626 [details] [review]
Bug 22246: Fix indexing of large fields with Elasticsearch

Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 4 Josef Moravec 2019-02-28 13:13:47 UTC
Created attachment 85863 [details] [review]
Bug 22246: Fix indexing of large fields with Elasticsearch

Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 5 Nick Clemens 2019-03-04 18:50:32 UTC
Please rebase
Comment 6 Ere Maijala 2019-03-05 09:13:37 UTC
Created attachment 86050 [details] [review]
Bug 22246: Fix indexing of large fields with Elasticsearch

Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense.

Test plan:
1. Reset ES mappings in administration
2. Check that sort is '0' for local-classification in biblio mappings.
3. Change sort back to '1' for local-classification for the next steps.
4. Create a record with 20 items, each with a 100 character long call number
5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>

Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Comment 7 Ere Maijala 2019-03-05 09:13:56 UTC
Rebased.
Comment 8 Nick Clemens 2019-03-07 21:23:20 UTC
Awesome work all!

Pushed to master for 19.05
Comment 9 Martin Renvoize 2019-03-12 08:47:18 UTC
Pushed to 18.11.x for 18.11.04
Comment 10 Lucas Gass 2019-03-15 17:19:32 UTC
not backporting to 18.05.x