|
Description
Dobrica Pavlinusic
2011-11-19 10:58:12 UTC
Created attachment 6345 [details] [review] rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 Created attachment 6347 [details] [review] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 Great idea. I'd personally prefer two options: --offset and --count, or, better, any mean to select an interval of biblionumbers, or specific biblionumbers... I was thinking about something like --where 'biblionumber > 100 and biblionumber < 200' which would allow precise selection of interesting range. (In reply to comment #4) > I was thinking about something like --where 'biblionumber > 100 and > biblionumber < 200' which would allow precise selection of interesting range. Or even, a --select option which would allow for example: --select "SELECT biblionumber FROM biblio WHERE frameworkcode='MONO'" --select "SELECT biblionumber FROM biblio WHERE datecreated > '2010' --select "SELECT biblionumber FROM biblioitems WHERE itemtype='MONO'" Created attachment 6994 [details] [review] [Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 http://bugs.koha-community.org/show_bug.cgi?id=7246 Signed-off-by: Liz Rea <wizzyrea@gmail.com> Verified that --limit 3 only exports 3 records of the indexing batch. I think this is a useful option for testers and would like to see it in. The other suggestions can be implemented at a later time. Created attachment 6995 [details] [review] [Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing Partial re-indexing is mostly useful when experimenting with Zebra configuration and your test database is too large to be re-indexed for each trial. I'm submitting it after I added this twice to source code of rebuild_zebra.pl and I hope it might prove useful to others also. Test scenario: 1. pick koha instance for which you don't mind truncating results in search 2. re-run rebuild_zebra.pl and add --limit 1000 http://bugs.koha-community.org/show_bug.cgi?id=7246 Signed-off-by: Liz Rea <wizzyrea@gmail.com> Verified that --limit 3 only exports 3 records of the indexing batch. I think this is a useful option for testers and would like to see it in. The other suggestions can be implemented at a later time. Created attachment 7200 [details] [review] Bug 7246 add min/ofset and WHERE options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * ofset = the ofset of record. Say 1000 to start rebuilding at the 1000th record of your database * min = how many records to export. Say 400 to export only 400 records * WHERE = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from ofset & min limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. _sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Comment on attachment 6995 [details] [review] [Signed Off] Bug 7246 - rebuild_zebra.pl --limit number for partial reindexing I've submitted another patch, that does more than your (it implement options suggested by Frederic for example). Obsoleting this one and switching back to needs signoff, liz or anyone else, thx for your feedback ! Created attachment 7379 [details] [review] Bug 7246 add min/offset and WHERE options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the ofset of record. Say 1000 to start rebuilding at the 1000th record of your database * min = how many records to export. Say 400 to export only 400 records * WHERE = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from ofset & min limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. _sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> The script works, but on further reflection it strikes me that the names of the new arguments do not make sense (I corrected "ofset" to "offset"): -offset implies the first record that's being processed, as does -min. Perhaps you could change the arguments to -offset and -length, or something else like that? Created attachment 7410 [details] [review] Bug 7246 add offset/length and where options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database * length = how many records to export. Say 400 to export only 400 records * where = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. (In reply to comment #11) > The script works, but on further reflection it strikes me that the names of the > new arguments do not make sense (I corrected "ofset" to "offset"): -offset > implies the first record that's being processed, as does -min. Perhaps you > could change the arguments to -offset and -length, or something else like that? You're right. our excuse is that we're french and our english is not always perfect ;-) New patch with offset/length and minor changes in documentation (--param instead of -param in the doc) Created attachment 7470 [details] [review] Bug 7246 add offset/length and where options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database * length = how many records to export. Say 400 to export only 400 records * where = add a where clause to rebuild only a given itemtype, or anything you want to filter on * l = how many items should be export with biblios. This is a usefull option when you have records with so many items that they can result in a record higher than 99999bytes, that zebra don't like at all Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> This patch adds the param for item_limit, as well as the documentation, but no actual support for it. Either the -l param and docs should be removed from this patch, and put in with another that fully implements, or the support for item_limit should be added to this patch. Created attachment 7564 [details] [review] Bug 7246 add offset/length and where options to rebuild_zebra This patch reimplement a feature that is on biblibre/master for Koha-community/master It adds 4 parameters: * offset = the offset of record. Say 1000 to start rebuilding at the 1000th record of your database * length = how many records to export. Say 400 to export only 400 records * where = add a where clause to rebuild only a given itemtype, or anything you want to filter on Another improvement resulting from offset & length limit is the rebuild_zebra_sliced.zsh that will be submitted in another patch. rebuild_zebra_sliced will slice your all database in small chunks, and, if something went wrong for a given slice, will slice the slice, and repeat, until you reach a slice size of 1, showing which record is wrong in your database. Signed-off-by: Jared Camins-Esakov <jcamins@cpbibliography.com> Removed mention of -l option for limiting number of items exported, as requested by QA manager. This can be re-added in a later patch. Thank you, Jared, for removing the unused param. We can add it back in at a later time. Marking Passed QA. Patch pushed. Chris_n, this is an ENH that can safely be added to 3.6 as it add a new feature and don't change anything to what is existing. However, unless it's properly documented, it may be missed by most ppl. So i've set it for 3.8, but if you want to commit it to 3.6, please add a comment here There have been no further reports of problems so I am marking this bug resolved. |