It seems that record deletions can cause extreme slowdowns for Koha installations with extremely large numbers of records. It would be helpful to be able to skip record deletions when processing the zebraqueue with rebuild_zebra.pl so the deletions can be processed with a lower frequency.
Created attachment 21493 [details] [review] Bug 10955 - Add ability to skip deletions in zebraqueue It seems that record deletions can cause extreme slowdowns for Koha installations with extremely large numbers of records. It would be helpful to be able to skip record deletions when processing the zebraqueue with rebuild_zebra.pl so the deletions can be processed with a lower frequency. Test Plan: 1) Disable any zebra indexing cronjobs you may have 2) Delete a record 3) Note the operation recordDelete in the zebraqueue table having done = 0 4) Run misc/migration_tools/rebuild_zebra.pl -b -z --skip-deletes 5) Note the delete still has done = 0 6) Run misc/migration_tools/rebuild_zebra.pl -b -z 7) Note the delete now has done = 1
Patch applied cleanly, go forth and signoff
Created attachment 22204 [details] Fast Add framework sample Sample Fast Add framework in use at VWML
Sorry, gremlins - that attachment went to the wrong bug!
Comment on attachment 22204 [details] Fast Add framework sample this was attached to the wrong bug!
Created attachment 24413 [details] [review] Bug 10955 - Add ability to skip deletions in zebraqueue It seems that record deletions can cause extreme slowdowns for Koha installations with extremely large numbers of records. It would be helpful to be able to skip record deletions when processing the zebraqueue with rebuild_zebra.pl so the deletions can be processed with a lower frequency. Test Plan: 1) Disable any zebra indexing cronjobs you may have 2) Delete a record 3) Note the operation recordDelete in the zebraqueue table having done = 0 4) Run misc/migration_tools/rebuild_zebra.pl -b -z --skip-deletes 5) Note the delete still has done = 0 6) Run misc/migration_tools/rebuild_zebra.pl -b -z 7) Note the delete now has done = 1 Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com>
Created attachment 24730 [details] [review] [PASSED QA] Bug 10955 - Add ability to skip deletions in zebraqueue It seems that record deletions can cause extreme slowdowns for Koha installations with extremely large numbers of records. It would be helpful to be able to skip record deletions when processing the zebraqueue with rebuild_zebra.pl so the deletions can be processed with a lower frequency. Test Plan: 1) Disable any zebra indexing cronjobs you may have 2) Delete a record 3) Note the operation recordDelete in the zebraqueue table having done = 0 4) Run misc/migration_tools/rebuild_zebra.pl -b -z --skip-deletes 5) Note the delete still has done = 0 6) Run misc/migration_tools/rebuild_zebra.pl -b -z 7) Note the delete now has done = 1 Signed-off-by: Jonathan Druart <jonathan.druart@biblibre.com> Signed-off-by: Katrin Fischer <Katrin.Fischer.83@web.de> Passes all tests and QA script. Also tested for authorities, no problems found.
Pushed to master. Thanks, Kyle! If the underlying issue in zebraidx never gets resolved, I think a better course for us in the long run is to introduce a soft-deletion mechanism -- basically, something like the current OPAC suppression mechanism, but which marks records as being logically deleted.
Pushed to 3.14.x, will be in 3.14.07