Bugzilla – Attachment 176768 Details for
Bug 38913
Elasticsearch indexing explodes with some oversized records with UTF-8 characters
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 38913: (QA follow-up) test UTF-8 exceptions in large MARC records
Bug-38913-QA-follow-up-test-UTF-8-exceptions-in-la.patch (text/plain), 2.08 KB, created by
David Cook
on 2025-01-20 03:03:29 UTC
(
hide
)
Description:
Bug 38913: (QA follow-up) test UTF-8 exceptions in large MARC records
Filename:
MIME Type:
Creator:
David Cook
Created:
2025-01-20 03:03:29 UTC
Size:
2.08 KB
patch
obsolete
>From 76c95cbc189cc7ba19d4fb77d99ff39ac88eff0e Mon Sep 17 00:00:00 2001 >From: David Cook <dcook@prosentient.com.au> >Date: Mon, 20 Jan 2025 02:55:12 +0000 >Subject: [PATCH] Bug 38913: (QA follow-up) test UTF-8 exceptions in large MARC > records > >MARC records with over 99999 bytes are invalid by spec, and when you use >UTF-8 encoded characters in your MARC records, there is the potential >to generate fatal errors in MARC::File::USMARC when it runs >"marc_to_utf8" from "MARC::File::Encode" during its "decode" operation. > >That is, if you MARC::File::USMARC->encode a MARC record >with over 99999 bytes (including a number of UTF-8 bytes), there >is the potential when you run MARC::File:USMARC->decode on that same >data that you'll generate a fatal exception. > >The main patch in bug 38913 wraps the function doing the decode, >so that a bad record doesn't crash processing. > >Without the patch, this unit test will fail. With the patch, this >unit test will pass. >--- > t/db_dependent/Koha/SearchEngine/Elasticsearch.t | 3 ++- > 1 file changed, 2 insertions(+), 1 deletion(-) > >diff --git a/t/db_dependent/Koha/SearchEngine/Elasticsearch.t b/t/db_dependent/Koha/SearchEngine/Elasticsearch.t >index 0abdf47001..3df796fb36 100755 >--- a/t/db_dependent/Koha/SearchEngine/Elasticsearch.t >+++ b/t/db_dependent/Koha/SearchEngine/Elasticsearch.t >@@ -16,6 +16,7 @@ > # along with Koha; if not, see <http://www.gnu.org/licenses>. > > use Modern::Perl; >+use Encode; > > use Test::More tests => 8; > use Test::Exception; >@@ -690,7 +691,7 @@ subtest 'Koha::SearchEngine::Elasticsearch::marc_records_to_documents () tests' > MARC::Field->new('999', '', '', c => '1234567'), > ); > >- my $item_field = MARC::Field->new('952', '', '', o => '123456789123456789123456789', p => '123456789', z => 'test'); >+ my $item_field = MARC::Field->new('952', '', '', o => '123456789123456789123456789', p => '123456789', z => Encode::decode('UTF-8','To naprawdÄ bardzo dÅuga notatka. MyÅlÄ, że bÄdzie sprawiaÄ kÅopoty.')); > my $items_count = 1638; > while(--$items_count) { > $large_marc_record->append_fields($item_field); >-- >2.39.5
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 38913
:
176698
|
176699
|
176701
|
176704
|
176767
| 176768