Bug 7246 - rebuild_zebra.pl --limit option to allow partial re-indexing
Summary: rebuild_zebra.pl --limit option to allow partial re-indexing
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: 3.8
Hardware: All All
: P2 enhancement (vote)
Target Milestone: ---
Assignee: Paul Poulain
QA Contact: Ian Walls
URL:
Whiteboard:
Keywords:
Depends on:
Blocks: 7286
  Show dependency treegraph
 
Reported: 2011-11-19 10:58 UTC by Dobrica Pavlinusic
Modified: 2013-12-05 20:04 UTC (History)
4 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: ---
Bot Control: ---
When did the bot last check this:
Who signed the patch off:
Text to go in the release notes:


Attachments
rebuild_zebra.pl --limit number for partial reindexing (1.83 KB, patch)
2011-11-19 11:02 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing (1.84 KB, patch)
2011-11-19 12:01 UTC, Dobrica Pavlinusic
Details | Diff | Splinter Review
[Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing (2.14 KB, patch)
2011-12-31 22:28 UTC, Liz Rea
Details | Diff | Splinter Review
[Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing (2.14 KB, patch)
2011-12-31 22:30 UTC, Liz Rea
Details | Diff | Splinter Review
Bug 7246 add min/ofset and WHERE options to rebuild_zebra (3.99 KB, patch)
2012-01-17 16:15 UTC, Paul Poulain
Details | Diff | Splinter Review
Bug 7246 add min/offset and WHERE options to rebuild_zebra (4.20 KB, patch)
2012-01-29 13:50 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7246 add offset/length and where options to rebuild_zebra (4.14 KB, patch)
2012-02-01 14:15 UTC, Paul Poulain
Details | Diff | Splinter Review
Bug 7246 add offset/length and where options to rebuild_zebra (4.37 KB, patch)
2012-02-07 13:37 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review
Bug 7246 add offset/length and where options to rebuild_zebra (3.90 KB, patch)
2012-02-10 15:15 UTC, Jared Camins-Esakov
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Dobrica Pavlinusic 2011-11-19 10:58:12 UTC
Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial.

I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also.
Comment 1 Dobrica Pavlinusic 2011-11-19 11:02:31 UTC Comment hidden (obsolete)
Comment 2 Dobrica Pavlinusic 2011-11-19 12:01:40 UTC Comment hidden (obsolete)
Comment 3 Frédéric Demians 2011-11-20 11:25:47 UTC
Great idea. I'd personally prefer two options: --offset and --count, or, better, any mean to select an interval of biblionumbers, or specific biblionumbers...
Comment 4 Dobrica Pavlinusic 2011-11-20 18:20:34 UTC
I was thinking about something like --where 'biblionumber > 100 and biblionumber < 200' which would allow precise selection of interesting range.
Comment 5 Frédéric Demians 2011-11-21 06:43:16 UTC
(In reply to comment #4)
> I was thinking about something like --where 'biblionumber > 100 and
> biblionumber < 200' which would allow precise selection of interesting range.

Or even, a --select option which would allow for example:

  --select "SELECT biblionumber FROM biblio WHERE frameworkcode='MONO'"
  --select "SELECT biblionumber FROM biblio WHERE datecreated > '2010'
  --select "SELECT biblionumber FROM biblioitems WHERE itemtype='MONO'"
Comment 6 Liz Rea 2011-12-31 22:28:39 UTC Comment hidden (obsolete)
Comment 7 Liz Rea 2011-12-31 22:30:28 UTC Comment hidden (obsolete)
Comment 8 Paul Poulain 2012-01-17 16:15:21 UTC Comment hidden (obsolete)
Comment 9 Paul Poulain 2012-01-17 16:16:49 UTC
Comment on attachment 6995 [details] [review]
[Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing

I've submitted another patch, that does more than your (it implement options suggested by Frederic for example). Obsoleting this one and switching back to needs signoff, liz or anyone else, thx for your feedback !
Comment 10 Jared Camins-Esakov 2012-01-29 13:50:57 UTC Comment hidden (obsolete)
Comment 11 Jared Camins-Esakov 2012-01-29 14:17:06 UTC
The script works, but on further reflection it strikes me that the names of the new arguments do not make sense (I corrected "ofset" to "offset"): -offset implies the first record that's being processed, as does -min. Perhaps you could change the arguments to -offset and -length, or something else like that?
Comment 12 Paul Poulain 2012-02-01 14:15:36 UTC Comment hidden (obsolete)
Comment 13 Paul Poulain 2012-02-01 14:17:46 UTC
(In reply to comment #11)
> The script works, but on further reflection it strikes me that the names of the
> new arguments do not make sense (I corrected "ofset" to "offset"): -offset
> implies the first record that's being processed, as does -min. Perhaps you
> could change the arguments to -offset and -length, or something else like that?

You're right. our excuse is that we're french and our english is not always perfect ;-)

New patch with offset/length and minor changes in documentation (--param instead of -param in the doc)
Comment 14 Jared Camins-Esakov 2012-02-07 13:37:49 UTC Comment hidden (obsolete)
Comment 15 Ian Walls 2012-02-07 15:38:42 UTC
This patch adds the param for item_limit, as well as the documentation, but no actual support for it.  Either the -l param and docs should be removed from this patch, and put in with another that fully implements, or the support for item_limit should be added to this patch.
Comment 16 Jared Camins-Esakov 2012-02-10 15:15:46 UTC
Created attachment 7564 [details] [review]
Bug 7246 add offset/length and where options to rebuild_zebra

This patch reimplement a feature that is on biblibre/master for Koha-community/master

It adds 4 parameters:
* offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database
* length = how many records to export. Say 400 to export only 400 records
* where = add a where clause to rebuild only a given itemtype, or anything you want to filter on

Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh
that will be submitted in another patch.
rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database.

Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com>
Removed mention of -l option for limiting number of items exported, as requested
by QA manager. This can be re-added in a later patch.
Comment 17 Ian Walls 2012-02-16 17:12:03 UTC
Thank you, Jared, for removing the unused param.  We can add it back in at a later time.

Marking Passed QA.
Comment 18 Paul Poulain 2012-02-17 10:01:37 UTC
Patch pushed. Chris_n, this is an ENH that can safely be added to 3.6 as it add a new feature and don't change anything to what is existing. However, unless it's properly documented, it may be missed by most ppl.
So i've set it for 3.8, but if you want to commit it to 3.6, please add a comment here
Comment 19 Jared Camins-Esakov 2012-12-31 00:38:08 UTC
There have been no further reports of problems so I am marking this bug resolved.