Bug 39104 - Elasticsearch indexing crashes with exception in catch block
Summary: Elasticsearch indexing crashes with exception in catch block
Status: RESOLVED DUPLICATE of bug 38913
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: 24.11
Hardware: All All
: P5 - low normal
Assignee: Bugs List
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-02-12 09:26 UTC by Simon Hohl
Modified: 2025-02-13 10:15 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Simon Hohl 2025-02-12 09:26:45 UTC
With ElasticSearch 8 I get the following error while indexing:

```
Can't locate object method "type" via package "UTF-8 "\xC3" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
" (perhaps you forgot to load "UTF-8 "\xC3" does not map to Unicode at /usr/share/perl5/MARC/File/Encode.pm line 35.
"?) at /usr/share/koha/bin/search_tools/rebuild_elasticsearch.pl line 321.
Something went wrong rebuilding indexes for ...
```

I am not used to perl, at first I thought I had some weird encoding in my record, but now I think this is actually a bug in the indexing script (rebuild_elasticsearch.pl line 321), which looks like this:

(...)
while ( my $record = $next->() ) {
    my $id     = $record->id // $record->authid;
    my $record = $record->record;
    $count++;
    if ( $verbose == 1 ) {
        _log( 1, "$count records processed\n" ) if ( $count % 1000 == 0);
    } else {
        _log( 2, "$id\n" );
    }

    push @id_buffer,     $id;
    push @commit_buffer, $record;
    if ( !( --$commit_count ) ) {
        _log( 1, "Committing $commit records...\n" );
        my $response;
        try{
            $response = $indexer->update_index( \@id_buffer, \@commit_buffer );
            _handle_response($response);
            _log( 1, "Commit complete\n" );
        } catch {
            _log(1,"Elasticsearch exception thrown: ".$_->type."\n");
            _log(2,"Details: ".$_->details."\n");
        };
        $commit_count  = $commit;
        @id_buffer     = ();
        @commit_buffer = ();
    }
}
(...)

So the catch log is blowing up the indexing process. I can get the script to process most records by changing those two logs in the catch-block, but I am unsure how to fix the logging to see what the underlying issue with my records is.
Comment 1 Janusz Kaczmarek 2025-02-12 14:48:44 UTC
At fist glance it looks very similar to the issue solved by Bug 38913. Could you please verify and maybe confirm that the above mentioned patch fixes your issue?
Comment 2 David Cook 2025-02-12 23:21:20 UTC
Yeah this sounds like it could be a duplicate of bug 38913
Comment 3 Katrin Fischer 2025-02-13 10:08:33 UTC
bug 38913 will be in the not yet released 24.11.02
Comment 4 Simon Hohl 2025-02-13 10:14:42 UTC
Yes, this is a duplicate. Thank you for pointing me toward the other ticket. :)
Comment 5 Simon Hohl 2025-02-13 10:15:04 UTC

*** This bug has been marked as a duplicate of bug 38913 ***