Bugzilla – Attachment 176698 Details for
Bug 38913
Elasticsearch indexing explodes with some oversized records with UTF-8 characters
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records
Bug-38913-bug-38416-follow-up-Elasticsearch-indexi.patch (text/plain), 2.25 KB, created by
Janusz Kaczmarek
on 2025-01-16 21:12:39 UTC
(
hide
)
Description:
Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records
Filename:
MIME Type:
Creator:
Janusz Kaczmarek
Created:
2025-01-16 21:12:39 UTC
Size:
2.25 KB
patch
obsolete
>From 400f234073ed4016b653f01c8c8d2507cb78ed93 Mon Sep 17 00:00:00 2001 >From: Janusz Kaczmarek <januszop@gmail.com> >Date: Thu, 16 Jan 2025 21:04:28 +0000 >Subject: [PATCH] Bug 38913: (bug 38416 follow-up) Elasticsearch indexing > explodes with oversized records > >After Bug 38416 Elasticsearch indexing explodes with oversized >records, especially with UTF encoded data. > >In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a >following snippet has been introduced: > >my $usmarc_record = $record->as_usmarc(); >my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); > >But if $record is oversized (> 99999 bytes), it is OK for MARC::Record >object, but not for $record->as_usmarc. The produced ISO 2709 string >is not correct and hence cannot be properly converted back to >MARC::Record object by new_from_usmarc. > >The result in this case can be like: > >UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. > >Since it is done without any eval / try, the whole reindex procedure >(for instance rebuild_elasticsearch.pl) is being randomly interrupted >with no explanation. > >Test plan: >========== >Hard to reproduce. But the explanation together with discussion in Bug >38416 (from 2024-12-15) explains and justifies the need of this added >eval. >--- > Koha/SearchEngine/Elasticsearch.pm | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > >diff --git a/Koha/SearchEngine/Elasticsearch.pm b/Koha/SearchEngine/Elasticsearch.pm >index 5faa3d13cc..6a40d70d93 100644 >--- a/Koha/SearchEngine/Elasticsearch.pm >+++ b/Koha/SearchEngine/Elasticsearch.pm >@@ -842,8 +842,9 @@ sub marc_records_to_documents { > my $usmarc_record = $record->as_usmarc(); > > #NOTE: Try to round-trip the record to prove it will work for retrieval after searching >- my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); >- if ( $decoded_usmarc_record->warnings() ) { >+ my $decoded_usmarc_record; >+ eval { $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); }; >+ if ( $@ || $decoded_usmarc_record->warnings() ) { > > #NOTE: We override the warnings since they're many and misleading > @warnings = ( >-- >2.39.5
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 38913
:
176698
|
176699
|
176701
|
176704
|
176767
|
176768