Bug 19502 - Result sets limited to 10000
Summary: Result sets limited to 10000
Status: Signed Off
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-10-20 15:37 UTC by Nick Clemens
Modified: 2018-06-19 13:20 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
Bug 19502: Limit pagination to first 10000 results when using ES (5.48 KB, patch)
2018-03-16 11:14 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19502: Limit pagination to first 10000 results when using ES (5.62 KB, patch)
2018-03-16 11:29 UTC, Séverine Queune
Details | Diff | Splinter Review
Bug 19520: (follow-up) Pass parameters to avoid making templates depend on search engine (5.13 KB, patch)
2018-04-19 23:15 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 19502: Retrieve index.max_result_window from ES (5.72 KB, patch)
2018-06-05 13:16 UTC, Julian Maurice
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2017-10-20 15:37:40 UTC
With the new feature to jump to a last page of search results it is apparent that we can't do this with our current use of ES. The standard limit is 10000 results, it can be increased at the cost of memory - need to look into 'Scroll' or 'Search after' options

https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules.html
 index.max_result_window
    The maximum value of from + size for searches to this index. Defaults to 10000. Search requests take heap memory and time proportional to from + size and this limits that memory. See Scroll or Search After for a more efficient alternative to raising this. 

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-request-scroll.html

https://www.elastic.co/guide/en/elasticsearch/reference/5.6/search-request-search-after.html
Comment 1 Nick Clemens 2018-03-16 11:14:58 UTC
Created attachment 73020 [details] [review]
Bug 19502: Limit pagination to first 10000 results when using ES

This patch is to avoid hitting an error page. We should eventually make the
max number returned configurable for ES.

To test:
1 - Have Koha running ES with 10,000+ records
2 - Search for '*'
3 - Click 'Last' to view last page of results
4 - 'Cannot perform search' error
5 - Apply patch
6 - Search again
7 - View 'Last' page
8 - No error, you go to the last of 10000
9 - Note the warning above the pagination buttons
Comment 2 Séverine Queune 2018-03-16 11:29:31 UTC
Created attachment 73024 [details] [review]
Bug 19502: Limit pagination to first 10000 results when using ES

This patch is to avoid hitting an error page. We should eventually make the
max number returned configurable for ES.

To test:
1 - Have Koha running ES with 10,000+ records
2 - Search for '*'
3 - Click 'Last' to view last page of results
4 - 'Cannot perform search' error
5 - Apply patch
6 - Search again
7 - View 'Last' page
8 - No error, you go to the last of 10000
9 - Note the warning above the pagination buttons

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 3 Séverine Queune 2018-03-16 11:30:23 UTC
Works as described for both staff client and opac.
Thanks Nick !
Comment 4 Nick Clemens 2018-04-19 23:15:08 UTC
Created attachment 74587 [details] [review]
Bug 19520: (follow-up) Pass parameters to avoid making templates depend on search engine
Comment 5 Julian Maurice 2018-06-05 13:10:37 UTC
I think we can do better than hardcoding the ES default value for max_result_window. We can retrieve the actual setting from ES, so that anyone can adjust this setting depending on their needs. I'm currently preparing a patch that will do that.
Comment 6 Julian Maurice 2018-06-05 13:16:45 UTC
Created attachment 75816 [details] [review]
Bug 19502: Retrieve index.max_result_window from ES

This avoid hardcoding '10000' in two different places and allow users to
adjust this setting.

Also, this patch fixes a bug when the search return less than 10000
results

Test plan:
1. Do a search that returns 10000+ records.
2. Note the warning above the pagination buttons
3. Go to the last page, no error
4. Change the ES setting:
   curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \
     '{"index": {"max_result_window": 20000}}'
5. Do another search that returns more than 10000 but less than 20000
6. Note that the warning does not show up
7. Go to the last page, still no error
Comment 7 Julian Maurice 2018-06-05 13:18:12 UTC
Nick, can you take a look at the last patch please ?
Comment 8 Alex Arnaud 2018-06-19 13:20:28 UTC
Comment on attachment 74587 [details] [review]
Bug 19520: (follow-up) Pass parameters to avoid making templates depend on search engine

Review of attachment 74587 [details] [review]:
-----------------------------------------------------------------

::: catalogue/search.pl
@@ +677,4 @@
>                                  last_page_offset => $last_page_offset,
>                                  previous_page_offset => $previous_page_offset) unless $pages < 2;
>              $template->param(   next_page_offset => $next_page_offset) unless $pages eq $current_page_number;
> +            warn "topage $hits_to_paginate";

Debug thing ?