|
Description
Stéphane Delaune
2011-12-02 08:41:05 UTC
Created attachment 6544 [details] [review] add rebuild_zebra_sliced.zsh (to rebuild zebra by batch automatically) add rebuild_zebra_sliced.zsh (to rebuild zebra by batch automatically) and its support in rebuild-zebra.pl just run misc/migration_tools/rebuild_zebra_sliced.zsh to test it Hi Stephane, can you add a bit more information about how this works and what it is doing/changing? It looks like a wrapper to the rebuild_zebra.pl script to filter the records being indexed by 'where' clauses or caount limits. Stéphane, rebuild_zebra_sliced relies on rebuild_zebra length and min options. Look at bug 7246, as i've been suggested to change the name or the variable, length/min were not good terms. + on biblibre/master min and length where/are inverted, i've reordered them correctly. rebuild_zebra_sliced must be updated (not a big deal) Removing "need signoff", as it can't be signed Created attachment 7463 [details] [review] the 7286 now depends of 7246 so it's the first of 2 patches the 7286 now depends of 7246 so it's the first of 2 patches Created attachment 7464 [details] [review] the second of 2 patches (its the 7286) the second of 2 patches (its the 7286) Created attachment 7465 [details] [review] all patches (7246 and 7286) into one file all patches (7246 and 7286) into one file Created attachment 7466 [details] [review] proposed patch the updated patch to merge after 7246 I proposed an updated patch (to merge after 7246). this script rebuild the zebra index recursively. he take 3 arguments : first specify how many records we must try at once second is the record number to start third is the record number to end the argumentys are optionnal (default value for the first is 10000, for the second is 0, and for the third is the las biblionumber in db) EXAMPLE to run the script on 800 000 biblios : ./reindex.zsh 50000 0 800000 will reindex the DB from biblionumber 0 to 800000, with chunks of 50k biblios or to reindexe all biblio step by step (10000 by 10000, it's the default values) : ./reindex.zsh in the last comment the script's name is wrong in examples : EXAMPLE to run the script on 800 000 biblios : ./misc/migration_tools/rebuild_zebra_sliced.zsh 50000 0 800000 will reindex the DB from biblionumber 0 to 800000, with chunks of 50k biblios or to reindexe all biblio step by step (10000 by 10000, it's the default values) : ./misc/migration_tools/rebuild_zebra_sliced.zsh the script can be launch from from any path (after exporting koha environment variables) Created attachment 7562 [details] [review] followup adding an option in rebuild_zebra_sliced.zsh Adds an optionnal fourth option called "TEMPDIRECTORY" to rebuild_zebra_sliced.zsh to define a temporary directory to rebuild_zebra (default value : /tmp/rebuild) Created attachment 7629 [details] [review] proposed patch (to merge after 7246) Created attachment 10095 [details] [review] Bug 7286 [Signed] Rebuild Zebra Sliced Works as expected, very useful for finding errant biblios that don't want to index. Minor comment, when running from command line some further feedback might be nice as currently if it's a slow index with large chunk size the screen is left blank for a long time leaving the user unsure whether anything is happening. Not enough to hold back sign-off. Does this script really need zsh? While I don't have anything against zsh as interactive shell, I think it's bad practice to introduce additional dependency to zsh for every Koha installation which want to use this script. I for one don't use zsh and don't have it installed on servers. Than again, this might be my Debian bias -- system scripts should use /bin/sh and not /bin/bash /bin/zfs, /bin/t?csh.... Created attachment 10598 [details] [review] Bug 7286: rebuild_zebra_sliced for biblios and authorities Complete rewrite of rebuild_zebra_sliced.zsh (renamed to .sh). Main improvements are: - both biblio and authority records are handled - records are exported only once It also add an option --skip-index to rebuild_zebra.pl that permit to use rebuild_zebra.pl as an 'export only' script. As the original patch is already signed-off, I do not obsolete it so that it could be pushed in master, but I think this new version could easily replace the old one. Created attachment 10621 [details] [review] Bug 7286: rebuild_zebra_sliced for biblios and authorities Fix some bugs: - add 'zebraidx commit' after each sucessful call to 'zebraidx update' - option -r is taken into account Created attachment 10655 [details] [review] Bug 7286: rebuild_zebra_sliced for biblios and authorities Complete rewrite of rebuild_zebra_sliced.zsh (renamed to .sh). Main improvements are: - both biblio and authority records are handled - records are exported only once It also add an option --skip-index to rebuild_zebra.pl that permit to use rebuild_zebra.pl as an 'export only' script. Description: Index Koha records by chunks. It is useful when some record causes errors and stop the indexation process. With this script, if indexation of one chunk fails, chunk is splitted in 2 (or 3) chunks, and indexation continue on these chunks. rebuild_zebra.pl is called only once to export records. Splitting and indexing is handled by this script (using yaz-marcdump and zebraidx). Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Passing QA on this... looks good and works well for me Patch pushed, It's safe and I think it's highly valuable to backport it to 3.8, as it's really a wonderful tool to index large DBs with a few invalid records ! It's a new script, so won't break anything This patch changes rebuild_zebra.pl as well, so not just a new script. I will do some testing to make sure that still works fine (it looks like it should) And will push to 3.8.x if it is ok. If you don't pass the new flag to rebuild_zebra.pl nothing changes, so it's a safe change. Pushed to 3.8.x, will be in 3.8.3 |