Elasticsearch number of results is by default limited by setting "index.max-result-window", default value is 10000. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window We use this setting : https://git.koha-community.org/Koha-community/Koha/src/commit/44d6528b566e5c16eba9b15f8aa24667293766c3/Koha/SearchEngine/Elasticsearch/Search.pm#L411 I propose we add this setting in index config.
This can be usefull for really big catalogs
Created attachment 135572 [details] [review] Bug 30882: Add max_result_window to index config Elasticsearch number of results is by default limited by setting "index.max-result-window", default value is 10000. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window We use this setting : https://git.koha-community.org/Koha-community/Koha/src/commit/44d6528b566e5c16eba9b15f8aa24667293766c3/Koha/SearchEngine/Elasticsearch/Search.pm#L411 I propose we add this setting in index config. Test plan : 1) Use Elasticsearch 2) Apply patch and flush memcached 3) Rebuild indexes : misc/search_tools/rebuild_elasticsearch.pl -v -b -d 4) Check the settings of index : curl '<cluster>:9200/<myindex>_biblios/_settings?pretty&filter_path=**.max_result_window' 5) You should see : "max_result_window" : "1000000"
Hi Fridolin. Using koha-testing-docker, how do I get step 4 to work? The curl command provided does not work for me. Thanks! David
(In reply to David Nind from comment #3) > Hi Fridolin. > > Using koha-testing-docker, how do I get step 4 to work? > > The curl command provided does not work for me. > > Thanks! > > David In KTD, use : curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.max_result_window'
(In reply to Fridolin Somers from comment #4) > In KTD, use : > curl > 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**. > max_result_window' Thanks Fridolin! That worked after adding this to the es section in koha-testing-docker's docker-compose.yml (after the networks configuration): ports: - "9200:9300" However, the result I got was: "max_result_window" : "10000" I think that is because this is the value in the patch (index.max_result_window: 10000)! Should that be index.max_result_window: 1000000? 8-)
Ahhh good catch. Could you fix the test plan when you push your patch ?
(In reply to Fridolin Somers from comment #6) > Ahhh good catch. > Could you fix the test plan when you push your patch ? No problem!
Created attachment 135751 [details] [review] Bug 30882: Add max_result_window to index config Elasticsearch number of results is by default limited by setting "index.max-result-window", default value is 10000. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window We use this setting: https://git.koha-community.org/Koha-community/Koha/src/commit/44d6528b566e5c16eba9b15f8aa24667293766c3/Koha/SearchEngine/Elasticsearch/Search.pm#L411 I propose we add this setting in index config. Test plan: 1) Use Elasticsearch 2) Apply patch and flush memcached 3) Rebuild indexes: misc/search_tools/rebuild_elasticsearch.pl -v -b -d 4) Check the settings of index (when using koha-testing-docker*): curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.max_result_window' 5) You should see: "max_result_window" : "1000000" * You also need to add this setting to the es section in koha-testing-docker's docker-compose.yml (after the networks configuration): ports: - "9200:9300" Signed-off-by: David Nind <david@davidnind.com>
I amended the patch to change 10000 to 1000000. I also amended the commit message to update the test plan - updated the curl statement and added koha-testing-docker information.
(In reply to David Nind from comment #9) > I amended the patch to change 10000 to 1000000. Thats a courageous change! I agree that 10K is too low.
(In reply to Marcel de Rooy from comment #10) > Thats a courageous change! Fridolin is the courageous one!
Created attachment 137445 [details] [review] Bug 30882: Add max_result_window to index config Elasticsearch number of results is by default limited by setting "index.max-result-window", default value is 10000. https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html#index-max-result-window We use this setting: https://git.koha-community.org/Koha-community/Koha/src/commit/44d6528b566e5c16eba9b15f8aa24667293766c3/Koha/SearchEngine/Elasticsearch/Search.pm#L411 I propose we add this setting in index config. Test plan: 1) Use Elasticsearch 2) Apply patch and flush memcached 3) Rebuild indexes: misc/search_tools/rebuild_elasticsearch.pl -v -b -d 4) Check the settings of index (when using koha-testing-docker*): curl 'es:9200/koha_kohadev_biblios/_settings?pretty&filter_path=**.max_result_window' 5) You should see: "max_result_window" : "1000000" * You also need to add this setting to the es section in koha-testing-docker's docker-compose.yml (after the networks configuration): ports: - "9200:9300" Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
I don't know what value should we have so I'm following Fridolin and Marcel ^^ Works, qa script happy, code looks good, passing QA :)
Pushed to master for 22.11. Nice work everyone, thanks!
Created attachment 137834 [details] [review] Bug 30882: (QA follow-up) Fix tests Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Backported to 22.05.x for 22.05.05
thx pushed to 21.11.x for 21.11.12
Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed. Nothing to document, marking resolved.