Attachment 114232 Details for Bug 26996 – Bug 26996: Convert Elasticsearch indexer commit buffer size to integer

[patch] Bug 26996: Convert Elasticsearch indexer commit buffer size to integer

Bug-26996-Convert-Elasticsearch-indexer-commit-buf.patch (text/plain), 1.74 KB, created by Joonas Kylmälä on 2020-12-07 13:44:09 UTC

(hide)

Description:

Filename:

MIME Type:

Creator: Joonas Kylmälä

Created: 2020-12-07 13:44:09 UTC

Size: 1.74 KB

patch

obsolete

>From d0a61fc8852200504835cc9cb154272548bc1522 Mon Sep 17 00:00:00 2001
>From: =?UTF-8?q?Bj=C3=B6rn=20Nyl=C3=A9n?= <bjorn.nylen@ub.lu.se>
>Date: Wed, 11 Nov 2020 08:35:21 +0100
>Subject: [PATCH] Bug 26996: Convert Elasticsearch indexer commit buffer size
> to integer
>MIME-Version: 1.0
>Content-Type: text/plain; charset=UTF-8
>Content-Transfer-Encoding: 8bit
>
>When multithreaded indexing is used, the commit size for children are spread
>out resulting in them being of type float. When records are processed and the
>commit counter decreased it may then never reach *exactly* 0. This means records
>are never commited. This patch makes sure the counter is an integer to avoid the
>problem.
>
>To test you must find a set of circumstances that causes the issue. For me:
>1. Run: ./rebuild_elasticsearch -v -b -p 2 -c 400
>2. Note that only one process is logging "Committing xxx records..."
>3. Kill processes.
>4. Apply patch.
>5. Repeat 1
>6. Note that both processes are logging "Committing xxx records..."
>
>Sponsored-by: Lund University Library
>Signed-off-by: Joonas KylmÃ¤lÃ¤ <joonas.kylmala@helsinki.fi>
>---
> misc/search_tools/rebuild_elasticsearch.pl | 1 +
> 1 file changed, 1 insertion(+)
>
>diff --git a/misc/search_tools/rebuild_elasticsearch.pl b/misc/search_tools/rebuild_elasticsearch.pl
>index ff83562f25..6b9487fcb5 100755
>--- a/misc/search_tools/rebuild_elasticsearch.pl
>+++ b/misc/search_tools/rebuild_elasticsearch.pl
>@@ -188,6 +188,7 @@ if ($slice_count > 1) {
>     }
>     # Fudge the commit count a bit to spread out the Elasticsearch commits
>     $commit *= 1 + 0.10 * $slice_index;
>+    $commit = int( $commit );
>     _log(1, "Processing slice @{[$slice_index + 1]} of $slice_count\n");
>     $iterator_options{slice} = { index => $slice_index, count => $slice_count };
> }
>-- 
>2.11.0

Actions: View | Diff | Splinter Review

Attachments on bug 26996: 113482 | 114227 | 114232 | 114679