Export.pl uses Koha::Biblioitems->search() to retrieve biblionumbers for the export process. The SQL code generated by this does a full dump of the the biblioitems table including the marcxml for each record. This table can be quite large and on smaller servers can actually cause debian to fire oom-killer on the process. As the results of the search are only used to push biblionumbers into an array it might make better sense to do a direct SQL query and not retrieve the marcxml column
Created attachment 53110 [details] [review] Bug 16855 - Poor performance due to high overhead of SQL call in export.pl This patch eliminates all columns retrieved in the biblioitems query except for biblionumber. To test: 1 - Go to tools->Export data 2 - Export using various filters and note you get expected records 3 - Apply patch 4 - Ensure exported results match results before patch
Created attachment 53122 [details] [review] [SIGNED-OFF] Bug 16855 - Poor performance due to high overhead of SQL call in export.pl This patch eliminates all columns retrieved in the biblioitems query except for biblionumber. To test: 1 - Go to tools->Export data 2 - Export using various filters and note you get expected records 3 - Apply patch 4 - Ensure exported results match results before patch Signed-off-by: Srdjan <srdjan@catalyst.net.nz>
Created attachment 53138 [details] [review] Bug 16855 - Poor performance due to high overhead of SQL call in export.pl This patch eliminates all columns retrieved in the biblioitems query except for biblionumber. To test: 1 - Go to tools->Export data 2 - Export using various filters and note you get expected records 3 - Apply patch 4 - Ensure exported results match results before patch Signed-off-by: Srdjan <srdjan@catalyst.net.nz>
Quick benchmark using: use Koha::Biblioitems; my @record_ids; my $biblioitems = Koha::Biblioitems->search( {}, { join => 'items', columns => 'biblionumber' } ); while ( my $biblioitem = $biblioitems->next ) { push @record_ids, $biblioitem->biblionumber; } and use Koha::Biblioitems; my @record_ids; my $biblioitems = Koha::Biblioitems->search( {}, { join => 'items' } ); while ( my $biblioitem = $biblioitems->next ) { push @record_ids, $biblioitem->biblionumber; } gives 1.4s vs 1.7s with 8600 items.
Pushed to master for Koha 16.11,t thanks Nick!
Pushed in 16.05. Will be in 16.05.02.
Patch pushed to 3.22.x, will be in 3.22.10