Bugzilla – Attachment 109241 Details for
Bug 26312
Add some error handling during Elasticsearch indexing
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 26312: Catch ES Client errors, log, and continue indexing when error encountered
Bug-26312-Catch-ES-Client-errors-log-and-continue-.patch (text/plain), 5.52 KB, created by
Nick Clemens
on 2020-08-27 16:09:57 UTC
(
hide
)
Description:
Bug 26312: Catch ES Client errors, log, and continue indexing when error encountered
Filename:
MIME Type:
Creator:
Nick Clemens
Created:
2020-08-27 16:09:57 UTC
Size:
5.52 KB
patch
obsolete
>From e63d13021ba1c04437f0154097c9a763898655e2 Mon Sep 17 00:00:00 2001 >From: Nick Clemens <nick@bywatersolutions.com> >Date: Thu, 27 Aug 2020 16:04:57 +0000 >Subject: [PATCH] Bug 26312: Catch ES Client errors, log, and continue indexing > when error encountered > >This catches a timeout response from the ES server, logs this, and continues the indexing > >To test: >1 - perl misc/search_tools/rebuild_elasticsearch.pl >2 - Make the ES server timeout (I don't have good instruction yet) >3 - Watch the job crash >4 - Apply patches >5 - perl misc/search_tools/rebuild_elasticsearch.pl >6 - Make the server timeout >7 - Note the job reports failed commit, and continues >--- > Koha/Exceptions/Elasticsearch.pm | 47 ++++++++++++++++++++++++++++-- > Koha/SearchEngine/Elasticsearch/Indexer.pm | 10 +++++++ > misc/search_tools/rebuild_elasticsearch.pl | 22 ++++++++++---- > 3 files changed, 70 insertions(+), 9 deletions(-) > >diff --git a/Koha/Exceptions/Elasticsearch.pm b/Koha/Exceptions/Elasticsearch.pm >index d4fd589fbf..7ad8538069 100644 >--- a/Koha/Exceptions/Elasticsearch.pm >+++ b/Koha/Exceptions/Elasticsearch.pm >@@ -1,5 +1,20 @@ > package Koha::Exceptions::Elasticsearch; > >+# This file is part of Koha. >+# >+# Koha is free software; you can redistribute it and/or modify it >+# under the terms of the GNU General Public License as published by >+# the Free Software Foundation; either version 3 of the License, or >+# (at your option) any later version. >+# >+# Koha is distributed in the hope that it will be useful, but >+# WITHOUT ANY WARRANTY; without even the implied warranty of >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the >+# GNU General Public License for more details. >+# >+# You should have received a copy of the GNU General Public License >+# along with Koha; if not, see <http://www.gnu.org/licenses>. >+ > use Modern::Perl; > > use Exception::Class ( >@@ -7,11 +22,37 @@ use Exception::Class ( > 'Koha::Exceptions::Elasticsearch' => { > description => 'Something went wrong!', > }, >+ 'Koha::Exceptions::Elasticsearch::BadResponse' => { >+ isa => 'Koha::Exceptions::Elasticsearch', >+ description => 'Bad response received when submitting request to Elasticsearch', >+ fields => [ 'error', 'details' ] >+ }, > 'Koha::Exceptions::Elasticsearch::MARCFieldExprParseError' => { >- isa => 'Koha::Exceptions::Elasticsearch', >- description => 'Parse error while processing MARC field expression in mapping', >+ isa => 'Koha::Exceptions::Elasticsearch', >+ description => 'Error parsing MARC Field Expression' > } >- > ); > >+=head1 NAME >+ >+Koha::Exceptions::Elasticsearch - Base class for elasticsearch exceptions >+ >+=head1 Exceptions >+ >+=head2 Koha::Exceptions::Elasticsearch >+ >+Generic Elasticsearch exception >+ >+=head2 Koha::Exceptions::Elasticsearch::BadResponse >+ >+Exception to be used when more a request to ES fails >+ >+=head2 Koha::Exceptions::Elasticsearch::MARCFieldExprParseError >+ >+Exception to be used when encountering an error parsing MARC Field Expression >+ >+=head1 Class methods >+ >+=cut >+ > 1; >diff --git a/Koha/SearchEngine/Elasticsearch/Indexer.pm b/Koha/SearchEngine/Elasticsearch/Indexer.pm >index e2c1c91fbd..ac71be1670 100644 >--- a/Koha/SearchEngine/Elasticsearch/Indexer.pm >+++ b/Koha/SearchEngine/Elasticsearch/Indexer.pm >@@ -25,6 +25,7 @@ use base qw(Koha::SearchEngine::Elasticsearch); > use Data::Dumper; > > use Koha::Exceptions; >+use Koha::Exceptions::Elasticsearch; > use C4::Context; > > =head1 NAME >@@ -111,12 +112,21 @@ sub update_index { > } > my $response; > if (@body) { >+ try{ > my $elasticsearch = $self->get_elasticsearch(); > $response = $elasticsearch->bulk( > index => $self->index_name, > type => 'data', # is just hard coded in Indexer.pm? > body => \@body > ); >+ } catch { >+ if( ref $_ eq 'Search::Elasticsearch::Error::Timeout' ){ >+ Koha::Exceptions::Elasticsearch::BadResponse->throw( >+ error => "Record commit failed.", >+ details => "Timeout", >+ ); >+ } >+ }; > } > return $response; > } >diff --git a/misc/search_tools/rebuild_elasticsearch.pl b/misc/search_tools/rebuild_elasticsearch.pl >index 6cff2eae2e..b4572be264 100755 >--- a/misc/search_tools/rebuild_elasticsearch.pl >+++ b/misc/search_tools/rebuild_elasticsearch.pl >@@ -116,6 +116,7 @@ use MARC::Field; > use MARC::Record; > use Modern::Perl; > use Pod::Usage; >+use Try::Tiny; > > my $verbose = 0; > my $commit = 5000; >@@ -288,12 +289,21 @@ sub _do_reindex { > push @commit_buffer, $record; > if ( !( --$commit_count ) ) { > _log( 1, "Committing $commit records...\n" ); >- my $response = $indexer->update_index( \@id_buffer, \@commit_buffer ); >- _handle_response($response); >- $commit_count = $commit; >- @id_buffer = (); >- @commit_buffer = (); >- _log( 1, "Commit complete\n" ); >+ my $response; >+ try{ >+ $response = $indexer->update_index( \@id_buffer, \@commit_buffer ); >+ _handle_response($response); >+ $commit_count = $commit; >+ @id_buffer = (); >+ @commit_buffer = (); >+ _log( 1, "Commit complete\n" ); >+ } catch { >+ if( ref $_ eq 'Koha::Exceptions::Elasticsearch::BadResponse'){ >+ _log(1,"Whoops"); >+ _log(1,$_->{error}); >+ _log(2,$_->{details}); >+ } >+ } > } > } > >-- >2.11.0
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 26312
:
109241
|
109626
|
113501
|
114229
|
117618
|
117619
|
119014
|
119079
|
119612
|
119613
|
119614
|
119615
|
119699
|
119700