Bug 27043 - Add to number_of_replicas and number_of_shards to index config
Summary: Add to number_of_replicas and number_of_shards to index config
Status: Pushed to oldoldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Fridolin Somers
QA Contact: Nick Clemens
URL:
Keywords:
: 25668 (view as bug list)
Depends on:
Blocks: 25439
  Show dependency treegraph
 
Reported: 2020-11-17 13:47 UTC by Fridolin Somers
Modified: 2021-02-11 22:32 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Text to go in the release notes:
Elasticsearch 6 server has default value 5 for "number_of_shards" but warn about Elasticsearch 7 having default value 1. So its is better to set this value in configuration file. Patch also sets number_of_replicas to 1. If you have only one Elasticsearch node, you have to set this value to 0.
Version(s) released in:
21.05.00,20.11.02,20.05.08,19.11.14


Attachments
Bug 27043: Add to number_of_replicas and number_of_shards to index config (1.52 KB, patch)
2020-11-17 14:01 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 27043: Add to number_of_replicas and number_of_shards to index config (1.58 KB, patch)
2020-12-20 19:48 UTC, Victor Grousset/tuxayo
Details | Diff | Splinter Review
Bug 27043: Add to number_of_replicas and number_of_shards to index config (1.63 KB, patch)
2020-12-21 14:32 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2020-11-17 13:47:24 UTC
With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.
Comment 1 Fridolin Somers 2020-11-17 14:01:03 UTC
Created attachment 113734 [details] [review]
Bug 27043: Add to number_of_replicas and number_of_shards to index config

With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.

Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
   curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
   "number_of_shards" : "5",
   "number_of_replicas" : "1"
Comment 2 Victor Grousset/tuxayo 2020-12-20 19:48:34 UTC
Created attachment 114544 [details] [review]
Bug 27043: Add to number_of_replicas and number_of_shards to index config

With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.

Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
   curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
   "number_of_shards" : "5",
   "number_of_replicas" : "1"

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 3 Victor Grousset/tuxayo 2020-12-20 19:49:27 UTC
Thanks Frido :D

Testing note: In koha-testing-docker, step 5) is
curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.number_of_*'
Comment 4 Nick Clemens 2020-12-21 14:32:41 UTC
Created attachment 114559 [details] [review]
Bug 27043: Add to number_of_replicas and number_of_shards to index config

With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.

Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
   curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
   "number_of_shards" : "5",
   "number_of_replicas" : "1"

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 5 Jonathan Druart 2021-01-04 13:10:43 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 6 Fridolin Somers 2021-01-07 13:50:55 UTC
Pushed to 20.11.x for 20.11.02
Comment 7 Andrew Fuerste-Henry 2021-01-11 21:09:02 UTC
Pushed to 20.05.x for 20.05.08
Comment 8 Victor Grousset/tuxayo 2021-01-18 18:30:06 UTC
*** Bug 25668 has been marked as a duplicate of this bug. ***
Comment 9 Victor Grousset/tuxayo 2021-01-18 18:35:32 UTC
Backported: Pushed to 19.11.x branch for 19.11.14