Summary: | Elasticsearch indexing needs a maximum length for `__sort` fields | ||
---|---|---|---|
Product: | Koha | Reporter: | Ere Maijala <ere.maijala> |
Component: | Searching - Elasticsearch | Assignee: | Ere Maijala <ere.maijala> |
Status: | CLOSED FIXED | QA Contact: | Josef Moravec <josef.moravec> |
Severity: | normal | ||
Priority: | P3 | CC: | josef.moravec, lucas, martin.renvoize, nick |
Version: | Main | ||
Hardware: | All | ||
OS: | All | ||
Change sponsored?: | --- | Patch complexity: | --- |
Documentation contact: | Documentation submission: | ||
Text to go in the release notes: | Version(s) released in: | ||
Attachments: |
Bug 22246: Fix indexing of large fields with Elasticsearch
Bug 22246: Fix indexing of large fields with Elasticsearch Bug 22246: Fix indexing of large fields with Elasticsearch Bug 22246: Fix indexing of large fields with Elasticsearch Bug 22246: Fix indexing of large fields with Elasticsearch |
Description
Ere Maijala
2019-01-31 12:22:42 UTC
Created attachment 84607 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Also fixes mapping for llevel while at it. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Created attachment 84608 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Created attachment 84626 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Created attachment 85863 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Please rebase Created attachment 86050 [details] [review] Bug 22246: Fix indexing of large fields with Elasticsearch Deduplicate multivalued fields and make sure sort fields are not excessively long. Also updates default mappings so that sort fields are not created for item fields where it doesn't make sense. Test plan: 1. Reset ES mappings in administration 2. Check that sort is '0' for local-classification in biblio mappings. 3. Change sort back to '1' for local-classification for the next steps. 4. Create a record with 20 items, each with a 100 character long call number 5. Check that when indexed, the record in ES does not have duplicates in any of the item fields and local-classification__sort is truncated to 255 characters. Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Josef Moravec <josef.moravec@gmail.com> Rebased. Awesome work all! Pushed to master for 19.05 Pushed to 18.11.x for 18.11.04 not backporting to 18.05.x |