Bug 7286 - Introduce rebuild_zebra_sliced.zsh to recursively rebuild zebra & be error proof
Summary: Introduce rebuild_zebra_sliced.zsh to recursively rebuild zebra & be error proof
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: 3.8
Hardware: All All
: P2 enhancement (vote)
Assignee: stephane.delaune
QA Contact: Mason James
URL:
Keywords:
Depends on: 7246
Blocks:
  Show dependency treegraph
 
Reported: 2011-12-02 08:41 UTC by stephane.delaune
Modified: 2013-12-05 19:59 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
add rebuild_zebra_sliced.zsh (to rebuild zebra by batch automatically) (9.90 KB, patch)
2011-12-02 17:20 UTC, stephane.delaune
Details | Diff | Splinter Review
the 7286 now depends of 7246 so it's the first of 2 patches (4.27 KB, patch)
2012-02-06 11:22 UTC, stephane.delaune
Details | Diff | Splinter Review
the second of 2 patches (its the 7286) (7.44 KB, patch)
2012-02-06 11:23 UTC, stephane.delaune
Details | Diff | Splinter Review
all patches (7246 and 7286) into one file (4.34 KB, patch)
2012-02-06 11:25 UTC, stephane.delaune
Details | Diff | Splinter Review
proposed patch (7.44 KB, patch)
2012-02-06 13:06 UTC, stephane.delaune
Details | Diff | Splinter Review
followup adding an option in rebuild_zebra_sliced.zsh (2.47 KB, patch)
2012-02-10 11:14 UTC, stephane.delaune
Details | Diff | Splinter Review
proposed patch (to merge after 7246) (6.84 KB, patch)
2012-02-13 16:04 UTC, stephane.delaune
Details | Diff | Splinter Review
Bug 7286 [Signed] Rebuild Zebra Sliced (6.94 KB, patch)
2012-06-10 13:06 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 7286: rebuild_zebra_sliced for biblios and authorities (11.09 KB, patch)
2012-07-02 12:22 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7286: rebuild_zebra_sliced for biblios and authorities (11.30 KB, patch)
2012-07-03 10:45 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 7286: rebuild_zebra_sliced for biblios and authorities (11.43 KB, patch)
2012-07-06 02:29 UTC, Mason James
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description stephane.delaune 2011-12-02 08:41:05 UTC

    
Comment 1 stephane.delaune 2011-12-02 17:20:40 UTC Comment hidden (obsolete)
Comment 2 stephane.delaune 2011-12-02 17:21:57 UTC
just run misc/migration_tools/rebuild_zebra_sliced.zsh to test it
Comment 3 Katrin Fischer 2011-12-21 07:23:43 UTC
Hi Stephane, 
can you add a bit more information about how this works and what it is doing/changing?
Comment 4 Tomás Cohen Arazi 2012-01-03 15:14:28 UTC
It looks like a wrapper to the rebuild_zebra.pl script to filter the records being indexed by 'where' clauses or caount limits.
Comment 5 Paul Poulain 2012-02-03 09:16:00 UTC
Stéphane, rebuild_zebra_sliced relies on rebuild_zebra length and min options.
Look at bug 7246, as i've been suggested to change the name or the variable, length/min were not good terms.

+ on biblibre/master min and length where/are inverted, i've reordered them correctly.
rebuild_zebra_sliced must be updated (not a big deal)

Removing "need signoff", as it can't be signed
Comment 6 stephane.delaune 2012-02-06 11:22:30 UTC Comment hidden (obsolete)
Comment 7 stephane.delaune 2012-02-06 11:23:41 UTC Comment hidden (obsolete)
Comment 8 stephane.delaune 2012-02-06 11:25:12 UTC Comment hidden (obsolete)
Comment 9 stephane.delaune 2012-02-06 13:06:44 UTC Comment hidden (obsolete)
Comment 10 stephane.delaune 2012-02-06 13:16:05 UTC
I proposed an updated patch (to merge after 7246).

this script rebuild the zebra index recursively.
he take 3 arguments :
first specify how many records we must try at once
second is the record number to start
third is the record number to end

the argumentys are optionnal (default value for the first is 10000, for the second is 0, and for the third is the las biblionumber in db)

EXAMPLE to run the script on 800 000 biblios :
./reindex.zsh 50000 0 800000
will reindex the DB from biblionumber 0 to 800000, with chunks of 50k biblios

or to reindexe all biblio step by step (10000 by 10000, it's the default values) :
./reindex.zsh
Comment 11 stephane.delaune 2012-02-06 13:25:20 UTC
in the last comment the script's name is wrong in examples :



EXAMPLE to run the script on 800 000 biblios :
./misc/migration_tools/rebuild_zebra_sliced.zsh 50000 0 800000
will reindex the DB from biblionumber 0 to 800000, with chunks of 50k biblios

or to reindexe all biblio step by step (10000 by 10000, it's the default
values) :
./misc/migration_tools/rebuild_zebra_sliced.zsh


the script can be launch from from any path (after exporting koha environment variables)
Comment 12 stephane.delaune 2012-02-10 11:14:53 UTC Comment hidden (obsolete)
Comment 13 stephane.delaune 2012-02-13 16:04:06 UTC Comment hidden (obsolete)
Comment 14 Martin Renvoize 2012-06-10 13:06:37 UTC Comment hidden (obsolete)
Comment 15 Dobrica Pavlinusic 2012-06-11 13:41:58 UTC
Does this script really need zsh? While I don't have anything against zsh as interactive shell, I think it's bad practice to introduce additional dependency to zsh for every Koha installation which want to use this script.

I for one don't use zsh and don't have it installed on servers. Than again, this might be my Debian bias -- system scripts should use /bin/sh and not /bin/bash /bin/zfs, /bin/t?csh....
Comment 16 Julian Maurice 2012-07-02 12:22:54 UTC Comment hidden (obsolete)
Comment 17 Julian Maurice 2012-07-03 10:45:35 UTC Comment hidden (obsolete)
Comment 18 Mason James 2012-07-06 02:29:14 UTC
Created attachment 10655 [details] [review]
Bug 7286: rebuild_zebra_sliced for biblios and authorities

Complete rewrite of rebuild_zebra_sliced.zsh (renamed to .sh). Main
improvements are:
  - both biblio and authority records are handled
  - records are exported only once

It also add an option --skip-index to rebuild_zebra.pl that permit to
use rebuild_zebra.pl as an 'export only' script.

Description:
Index Koha records by chunks. It is useful when some record causes
errors and stop the indexation process. With this script, if indexation
of one chunk fails, chunk is splitted in 2 (or 3) chunks, and
indexation continue on these chunks.
rebuild_zebra.pl is called only once to export records.
Splitting and indexing is handled by this script (using yaz-marcdump and
zebraidx).

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 19 Mason James 2012-07-06 02:33:33 UTC
Passing QA on this...

looks good and works well for me
Comment 20 Paul Poulain 2012-07-06 13:09:56 UTC
Patch pushed,
It's safe and I think it's highly valuable to backport it to 3.8, as it's really a wonderful tool to index large DBs with a few invalid records !
It's a new script, so won't break anything
Comment 21 Chris Cormack 2012-07-07 00:04:21 UTC
This patch changes rebuild_zebra.pl as well, so not just a new script. I will do some testing to make sure that still works fine (it looks like it should)

And will push to 3.8.x if it is ok.
Comment 22 Chris Cormack 2012-07-07 03:08:52 UTC
If you don't pass the new flag to rebuild_zebra.pl nothing changes, so it's a safe change.

Pushed to 3.8.x, will be in 3.8.3