Bugzilla – Attachment 176701 Details for
Bug 38913
Elasticsearch indexing explodes with some oversized records with UTF-8 characters
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records
Bug-38913-bug-38416-follow-up-Elasticsearch-indexi.patch (text/plain), 3.27 KB, created by
Janusz Kaczmarek
on 2025-01-16 22:21:40 UTC
(
hide
)
Description:
Bug 38913: (bug 38416 follow-up) Elasticsearch indexing explodes with oversized records
Filename:
MIME Type:
Creator:
Janusz Kaczmarek
Created:
2025-01-16 22:21:40 UTC
Size:
3.27 KB
patch
obsolete
>From c1fdd370ff78f08168bf8de69f1a44e22977c873 Mon Sep 17 00:00:00 2001 >From: Janusz Kaczmarek <januszop@gmail.com> >Date: Thu, 16 Jan 2025 21:04:28 +0000 >Subject: [PATCH] Bug 38913: (bug 38416 follow-up) Elasticsearch indexing > explodes with oversized records > >After Bug 38416 Elasticsearch indexing explodes with oversized >records, especially with UTF encoded data. > >In Koha::SearchEngine::Elasticsearch::marc_records_to_documents a >following snippet has been introduced: > >my $usmarc_record = $record->as_usmarc(); >my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); > >But if $record is oversized (> 99999 bytes), it is OK for MARC::Record >object, but not for $record->as_usmarc. The produced ISO 2709 string >is not correct and hence cannot be properly converted back to >MARC::Record object by new_from_usmarc. > >The result in this case can be like: > >UTF-8 "\x85" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. > >Since it is done without any eval / try, the whole reindex procedure >(for instance rebuild_elasticsearch.pl) is being randomly interrupted >with no explanation. > >Test plan: >========== >Hard to reproduce. But the explanation together with discussion in Bug >38416 (from 2024-12-15) explains and justifies the need of this added >eval. > >1. Have a standard KTD installation with Elasticsearch. >2. Use the provided test record - add it to Koha with > ./misc/migration_tools/bulkmarcimport.pl -b -file test.xml -m=MARCXML > (have patience). > During load process you should see a message like: > UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. >3. The record should get biblionumber 439. Check in librarian interface with > http://<your_addreess>:8081/cgi-bin/koha/catalogue/detail.pl?biblionumber=439 > that the record has been imported. > However, you should not be able to make a search for this record. >4. Try to reindex with: > ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 > You should get a message like: > UTF-8 "\xC4" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35. > Again, no search results. >5. Apply the patch ; restart_all. >6. Repeat reindex with: > ./misc/search_tools/rebuild_elasticsearch.pl -b -bn 439 > There should be no warning now and you should be able to find the record. >--- > Koha/SearchEngine/Elasticsearch.pm | 5 +++-- > 1 file changed, 3 insertions(+), 2 deletions(-) > >diff --git a/Koha/SearchEngine/Elasticsearch.pm b/Koha/SearchEngine/Elasticsearch.pm >index 5faa3d13cc..6a40d70d93 100644 >--- a/Koha/SearchEngine/Elasticsearch.pm >+++ b/Koha/SearchEngine/Elasticsearch.pm >@@ -842,8 +842,9 @@ sub marc_records_to_documents { > my $usmarc_record = $record->as_usmarc(); > > #NOTE: Try to round-trip the record to prove it will work for retrieval after searching >- my $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); >- if ( $decoded_usmarc_record->warnings() ) { >+ my $decoded_usmarc_record; >+ eval { $decoded_usmarc_record = MARC::Record->new_from_usmarc($usmarc_record); }; >+ if ( $@ || $decoded_usmarc_record->warnings() ) { > > #NOTE: We override the warnings since they're many and misleading > @warnings = ( >-- >2.39.5
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 38913
:
176698
|
176699
|
176701
|
176704
|
176767
|
176768