From 400f234073ed4016b653f01c8c8d2507cb78ed93 Mon Sep 17 00:00:00 2001 From: Janusz Kaczmarek Date: Thu, 16 Jan 2025 21:04:28 +0000 Subject: [PATCH] Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records After Bug 38416 Elasticsearch indexing explodes with oversized records, especially with UTF encoded data. In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a following snippet has been introduced: my $usmarc_record = $record->as_usmarc(); my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); But if $record is oversized (> 99999 bytes), it is OK for MARC::Record object, but not for $record->as_usmarc. The produced ISO 2709 string is not correct and hence cannot be properly converted back to MARC::Record object by new_from_usmarc. The result in this case can be like: UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. Since it is done without any eval / try, the whole reindex procedure (for instance rebuild_elasticsearch.pl) is being randomly interrupted with no explanation. Test plan: ========== Hard to reproduce. But the explanation together with discussion in Bug 38416 (from 2024-12-15) explains and justifies the need of this added eval. --- Koha/SearchEngine/Elasticsearch.pm | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/Koha/SearchEngine/Elasticsearch.pm b/Koha/SearchEngine/Elasticsearch.pm index 5faa3d13cc..6a40d70d93 100644 --- a/Koha/SearchEngine/Elasticsearch.pm +++ b/Koha/SearchEngine/Elasticsearch.pm @@ -842,8 +842,9 @@ sub marc_records_to_documents { my $usmarc_record = $record->as_usmarc(); #NOTE: Try to round-trip the record to prove it will work for retrieval after searching - my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); - if ( $decoded_usmarc_record->warnings() ) { + my $decoded_usmarc_record; + eval { $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); }; + if ( $@ || $decoded_usmarc_record->warnings() ) { #NOTE: We override the warnings since they're many and misleading @warnings = ( -- 2.39.5