Currently there's no limit to the length of *__sort fields. If such field is created from item information or something else that can be repeated a lot, the field may become too long. Such field is e.g. local-classification, which is by default created from 952o. If there are hundreds or thousands of items, the field may become longer than what Elasticsearch accepts. Also consider making it a non-sortable field by default.
Created attachment 84607 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Also fixes mapping for llevel while at it. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Created attachment 84608 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters.
Created attachment 84626 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 85863 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Please rebase
Created attachment 86050 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com>
Rebased.
Awesome work all! Pushed to master for 19.05
Pushed to 18.11.x for 18.11.04
not backporting to 18.05.x