Bug 27043

Summary: Add to number_of_replicas and number_of_shards to index config
Product: Koha Reporter: Fridolin Somers <fridolin.somers>
Component: Searching - ElasticsearchAssignee: Fridolin Somers <fridolin.somers>
Status: CLOSED FIXED QA Contact: Nick Clemens <nick>
Severity: normal    
Priority: P5 - low CC: andrewfh, caroline.cyr-la-rose, jonathan.druart, nick, victor
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=25668
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=30882
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Elasticsearch 6 server has default value 5 for "number_of_shards" but warn about Elasticsearch 7 having default value 1. So its is better to set this value in configuration file. Patch also sets number_of_replicas to 1. If you have only one Elasticsearch node, you have to set this value to 0.
Version(s) released in:
21.05.00,20.11.02,20.05.08,19.11.14
Bug Depends on:    
Bug Blocks: 25439    
Attachments: Bug 27043: Add to number_of_replicas and number_of_shards to index config
Bug 27043: Add to number_of_replicas and number_of_shards to index config
Bug 27043: Add to number_of_replicas and number_of_shards to index config

Description Fridolin Somers 2020-11-17 13:47:24 UTC
With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.
Comment 1 Fridolin Somers 2020-11-17 14:01:03 UTC
Created attachment 113734 [details] [review]
Bug 27043: Add to number_of_replicas and number_of_shards to index config

With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.

Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
   curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
   "number_of_shards" : "5",
   "number_of_replicas" : "1"
Comment 2 Victor Grousset/tuxayo 2020-12-20 19:48:34 UTC
Created attachment 114544 [details] [review]
Bug 27043: Add to number_of_replicas and number_of_shards to index config

With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.

Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
   curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
   "number_of_shards" : "5",
   "number_of_replicas" : "1"

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 3 Victor Grousset/tuxayo 2020-12-20 19:49:27 UTC
Thanks Frido :D

Testing note: In koha-testing-docker, step 5) is
curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.number_of_*'
Comment 4 Nick Clemens 2020-12-21 14:32:41 UTC
Created attachment 114559 [details] [review]
Bug 27043: Add to number_of_replicas and number_of_shards to index config

With Elasticsearch 6 (>6.4), we have a warning on index creation :
  the default number of shards will change from [5] to [1] in 7.0.0

See https://github.com/elastic/elasticsearch/pull/30587

I propose to add number_of_shards in index config.

Also add number_of_replicas that is better explicit.
In case on only one node, it must be 0.

Test plan :
1) Use Elasticsearch
2) Apply patch and flush memcached
3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d
4) Check you dont have a warning about number of shards
5) Check the settings of index :
   curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.number_of_*'
6) You should see :
   "number_of_shards" : "5",
   "number_of_replicas" : "1"

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 5 Jonathan Druart 2021-01-04 13:10:43 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 6 Fridolin Somers 2021-01-07 13:50:55 UTC
Pushed to 20.11.x for 20.11.02
Comment 7 Andrew Fuerste-Henry 2021-01-11 21:09:02 UTC
Pushed to 20.05.x for 20.05.08
Comment 8 Victor Grousset/tuxayo 2021-01-18 18:30:06 UTC
*** Bug 25668 has been marked as a duplicate of this bug. ***
Comment 9 Victor Grousset/tuxayo 2021-01-18 18:35:32 UTC
Backported: Pushed to 19.11.x branch for 19.11.14