Bugzilla – Attachment 76796 Details for
Bug 19502
Result sets limited to 10000
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 19502: Retrieve index.max_result_window from ES
Bug-19502-Retrieve-indexmaxresultwindow-from-ES.patch (text/plain), 5.80 KB, created by
Alex Arnaud
on 2018-07-09 13:32:16 UTC
(
hide
)
Description:
Bug 19502: Retrieve index.max_result_window from ES
Filename:
MIME Type:
Creator:
Alex Arnaud
Created:
2018-07-09 13:32:16 UTC
Size:
5.80 KB
patch
obsolete
>From 9828e591cee161d818af53002843f176dcf72b58 Mon Sep 17 00:00:00 2001 >From: Julian Maurice <julian.maurice@biblibre.com> >Date: Tue, 5 Jun 2018 15:11:34 +0200 >Subject: [PATCH] Bug 19502: Retrieve index.max_result_window from ES > >This avoid hardcoding '10000' in two different places and allow users to >adjust this setting. > >Also, this patch fixes a bug when the search return less than 10000 >results > >Test plan: >1. Do a search that returns 10000+ records. >2. Note the warning above the pagination buttons >3. Go to the last page, no error >4. Change the ES setting: > curl -XPUT http://elasticsearch/koha_master_biblios/_settings -d \ > '{"index": {"max_result_window": 20000}}' >5. Do another search that returns more than 10000 but less than 20000 >6. Note that the warning does not show up >7. Go to the last page, still no error > >Signed-off-by: Nick Clemens <nick@bywatersolutions.com> >Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com> >--- > Koha/SearchEngine/Elasticsearch/Search.pm | 19 +++++++++++++++++++ > Koha/SearchEngine/Zebra/Search.pm | 2 ++ > catalogue/search.pl | 3 ++- > opac/opac-search.pl | 3 ++- > .../Koha_SearchEngine_Elasticsearch_Search.t | 15 +++++++++++++-- > 5 files changed, 38 insertions(+), 4 deletions(-) > >diff --git a/Koha/SearchEngine/Elasticsearch/Search.pm b/Koha/SearchEngine/Elasticsearch/Search.pm >index 063dca3..6b2f2a9 100644 >--- a/Koha/SearchEngine/Elasticsearch/Search.pm >+++ b/Koha/SearchEngine/Elasticsearch/Search.pm >@@ -399,6 +399,25 @@ sub json2marc { > return $marc; > } > >+sub max_result_window { >+ my ($self) = @_; >+ >+ $self->store( >+ Catmandu::Store::ElasticSearch->new(%{ $self->get_elasticsearch_params }) >+ ) unless $self->store; >+ >+ my $index_name = $self->store->index_name; >+ my $settings = $self->store->es->indices->get_settings( >+ index => $index_name, >+ params => { include_defaults => 1, flat_settings => 1 }, >+ ); >+ >+ my $max_result_window = $settings->{$index_name}->{settings}->{'index.max_result_window'}; >+ $max_result_window //= $settings->{$index_name}->{defaults}->{'index.max_result_window'}; >+ >+ return $max_result_window; >+} >+ > =head2 _convert_facets > > my $koha_facets = _convert_facets($es_facets, $expanded_facet); >diff --git a/Koha/SearchEngine/Zebra/Search.pm b/Koha/SearchEngine/Zebra/Search.pm >index 2fb45da..45c050c 100644 >--- a/Koha/SearchEngine/Zebra/Search.pm >+++ b/Koha/SearchEngine/Zebra/Search.pm >@@ -110,4 +110,6 @@ sub search_auth_compat { > C4::AuthoritiesMarc::SearchAuthorities(@params); > } > >+sub max_result_window { undef } >+ > 1; >diff --git a/catalogue/search.pl b/catalogue/search.pl >index 633200d..18b22b3 100755 >--- a/catalogue/search.pl >+++ b/catalogue/search.pl >@@ -619,7 +619,8 @@ for (my $i=0;$i<@servers;$i++) { > ## FIXME: add a global function for this, it's better than the current global one > ## Build the page numbers on the bottom of the page > my @page_numbers; >- my $hits_to_paginate = C4::Context->preference('SearchEngine') eq 'Elasticsearch' ? 10000 : $hits; >+ my $max_result_window = $searcher->max_result_window; >+ my $hits_to_paginate = ($max_result_window && $max_result_window < $hits) ? $max_result_window : $hits; > $template->param( hits_to_paginate => $hits_to_paginate ); > # total number of pages there will be > my $pages = ceil($hits_to_paginate / $results_per_page); >diff --git a/opac/opac-search.pl b/opac/opac-search.pl >index 18dd155..fbc0c30 100755 >--- a/opac/opac-search.pl >+++ b/opac/opac-search.pl >@@ -830,7 +830,8 @@ for (my $i=0;$i<@servers;$i++) { > } > ## Build the page numbers on the bottom of the page > my @page_numbers; >- my $hits_to_paginate = C4::Context->preference('SearchEngine') eq 'Elasticsearch' ? 10000 : $hits; >+ my $max_result_window = $searcher->max_result_window; >+ my $hits_to_paginate = ($max_result_window && $max_result_window < $hits) ? $max_result_window : $hits; > $template->param( hits_to_paginate => $hits_to_paginate ); > # total number of pages there will be > my $pages = ceil($hits_to_paginate / $results_per_page); >diff --git a/t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t b/t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t >index 5085690..7466f07 100644 >--- a/t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t >+++ b/t/db_dependent/Koha_SearchEngine_Elasticsearch_Search.t >@@ -17,10 +17,12 @@ > > use Modern::Perl; > >-use Test::More tests => 13; >+use Test::More tests => 15; > use t::lib::Mocks; > > use Koha::SearchEngine::Elasticsearch::QueryBuilder; >+use Koha::SearchEngine::Elasticsearch::Indexer; >+ > > my $builder = Koha::SearchEngine::Elasticsearch::QueryBuilder->new( { index => 'mydb' } ); > >@@ -41,9 +43,11 @@ SKIP: { > > eval { $builder->get_elasticsearch_params; }; > >- skip 'ElasticSeatch configuration not available', 6 >+ skip 'ElasticSeatch configuration not available', 8 > if $@; > >+ Koha::SearchEngine::Elasticsearch::Indexer->new({ index => 'mydb' })->drop_index; >+ > ok( my $results = $searcher->search( $query) , 'Do a search ' ); > > ok( my $marc = $searcher->json2marc( $results->first ), 'Convert JSON to MARC'); >@@ -56,6 +60,13 @@ SKIP: { > > is ( $count = $searcher->count_auth_use($searcher,1), 0, 'Testing count_auth_use'); > >+ is ($searcher->max_result_window, 10000, 'By default, max_result_window is 10000'); >+ $searcher->store->es->indices->put_settings(index => $searcher->store->index_name, body => { >+ 'index' => { >+ 'max_result_window' => 12000, >+ }, >+ }); >+ is ($searcher->max_result_window, 12000, 'max_result_window returns the correct value'); > } > > subtest 'json2marc' => sub { >-- >2.7.4
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 19502
:
73020
|
73024
|
74587
|
75816
|
76371
|
76372
|
76788
|
76789
|
76790
|
76791
|
76792
|
76794
|
76795
| 76796 |
76798
|
76799