We can speed up reindexing Zebra when we use parallel exporting, creating multiple export files in the same directory. This directory is passed later to zebraidx.
Created attachment 174418 [details] [review] Bug 38408: Add parallel exporting in rebuild_zebra.pl The first part of the Zebra rebuild is the exporting. This part is made faster. The second part with zebraidx is not changed. A new commandline parameter -forks is added to the rebuild_zebra.pl script. A subroutine export_marc_records is added between index_records and export_marc_records_from_sth. The last routine has a new parameter, the sequence number of the export file. NOTE: This report does not touch koha-rebuild-zebra yet! This will be done on a follow-up. Test plan: Note that the number of forks/records below can be adjusted according to your server and database setup. [1] Reindex a subset of 100 records without forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -r -d /tmp/rebuild01 -k --length 100 Check if /tmp/rebuild01/biblio contains one export file for auth/bib. Verify that max. 100 auth and bib were indexed (check Auth search, Cataloguing) [2] Reindex an additional subset of 100 recs with forks (remove -r, add -forks): su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild02 -k --length 100 --offset 100 -forks 3 Check if /tmp/rebuild02/biblio contains 3 export files for auth/bib. Verify that max. 200 auth and bib were indexed (check Auth search, Cataloguing) [3] Run a full reindex with forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild03 -k -forks 3 Check both searches again. [4] Bonus: To get a feeling of improved speed, reindex a larger production db with and without using -forks. (Use something like above.) You may add -I to skip indexing in order to better compare both exports. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Reindexed a prod db in 96 mins instead of 150 mins (3 forks, 4 cores). Main gain in biblio export; complete export took 35 mins, zebraidx 61 mins.
(In reply to Marcel de Rooy from comment #1) > NOTE: This report does not touch koha-rebuild-zebra yet! This will be > done on a follow-up. Planned on bug 38427
Note for QA: A few smaller improvements are reported on bug 38427 as well.
This makes a lot of sense conceptually. I can't believe we didn't do this years ago. I've got a few questions and comments though: 1. Is there are a reason you moved around a bunch of the top "use" statements? It makes it harder to see what's actually changed at the top. 2. There's a 'my $chunk_size = 100000;' at the top of the script which appears to be unused? 3. It wouldn't hurt to add some more code comments to make it easier to read/understand what's happening. Not sure that I follow the math when just reading the code. I would've expected chunk_size to stay fixed and $num_records_exported to be the number of records actually exported. Not sure this all adds up... but don't have time to test right now.
(In reply to David Cook from comment #4) > 1. Is there are a reason you moved around a bunch of the top "use" > statements? It makes it harder to see what's actually changed at the top. It was kind of disorganized, and perltidy was also complaining. So I rearranged a bit more than strictly needed. But nothing special. > 2. There's a 'my $chunk_size = 100000;' at the top of the script which > appears to be unused? True. See the follow-up bug. Just a remainder of something obsoleted during dev. > 3. It wouldn't hurt to add some more code comments to make it easier to > read/understand what's happening. Not sure that I follow the math when just > reading the code. I would've expected chunk_size to stay fixed and > $num_records_exported to be the number of records actually exported. Not > sure this all adds up... but don't have time to test right now. I tried to put some extra comments in the main loop. The tricky thing is that the loop inside is normally the child, but could be the parent too if you dont fork. I am using chunk_size to control the loop; so it is adjusted before the last run. Added a TODO on the follow-up bug about passing data from child back to parent.
Created attachment 178291 [details] [review] Bug 38408: Add parallel exporting in rebuild_zebra.pl The first part of the Zebra rebuild is the exporting. This part is made faster. The second part with zebraidx is not changed. A new commandline parameter -forks is added to the rebuild_zebra.pl script. A subroutine export_marc_records is added between index_records and export_marc_records_from_sth. The last routine has a new parameter, the sequence number of the export file. NOTE: This report does not touch koha-rebuild-zebra yet! This will be done on a follow-up. Test plan: Note that the number of forks/records below can be adjusted according to your server and database setup. [1] Reindex a subset of 100 records without forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -r -d /tmp/rebuild01 -k --length 100 Check if /tmp/rebuild01/biblio contains one export file for auth/bib. Verify that max. 100 auth and bib were indexed (check Auth search, Cataloguing) [2] Reindex an additional subset of 100 recs with forks (remove -r, add -forks): su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild02 -k --length 100 --offset 100 -forks 3 Check if /tmp/rebuild02/biblio contains 3 export files for auth/bib. Verify that max. 200 auth and bib were indexed (check Auth search, Cataloguing) [3] Run a full reindex with forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild03 -k -forks 3 Check both searches again. [4] Bonus: To get a feeling of improved speed, reindex a larger production db with and without using -forks. (Use something like above.) You may add -I to skip indexing in order to better compare both exports. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Reindexed a prod db in 96 mins instead of 150 mins (3 forks, 4 cores). Main gain in biblio export; complete export took 35 mins, zebraidx 61 mins.
Rebased for tidy codebase stuff
Going to look at testing this one today...
(In reply to David Cook from comment #8) > Going to look at testing this one today... Sorry Marcel. Ran out of time this week...
(In reply to David Cook from comment #9) > (In reply to David Cook from comment #8) > > Going to look at testing this one today... > > Sorry Marcel. Ran out of time this week... Too bad :) New week, new chances..
Created attachment 178814 [details] [review] Bug 38408: Add parallel exporting in rebuild_zebra.pl The first part of the Zebra rebuild is the exporting. This part is made faster. The second part with zebraidx is not changed. A new commandline parameter -forks is added to the rebuild_zebra.pl script. A subroutine export_marc_records is added between index_records and export_marc_records_from_sth. The last routine has a new parameter, the sequence number of the export file. NOTE: This report does not touch koha-rebuild-zebra yet! This will be done on a follow-up. Test plan: Note that the number of forks/records below can be adjusted according to your server and database setup. [1] Reindex a subset of 100 records without forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -r -d /tmp/rebuild01 -k --length 100 Check if /tmp/rebuild01/biblio contains one export file for auth/bib. Verify that max. 100 auth and bib were indexed (check Auth search, Cataloguing) [2] Reindex an additional subset of 100 recs with forks (remove -r, add -forks): su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild02 -k --length 100 --offset 100 -forks 3 Check if /tmp/rebuild02/biblio contains 3 export files for auth/bib. Verify that max. 200 auth and bib were indexed (check Auth search, Cataloguing) [3] Run a full reindex with forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild03 -k -forks 3 Check both searches again. [4] Bonus: To get a feeling of improved speed, reindex a larger production db with and without using -forks. (Use something like above.) You may add -I to skip indexing in order to better compare both exports. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Reindexed a prod db in 96 mins instead of 150 mins (3 forks, 4 cores). Main gain in biblio export; complete export took 35 mins, zebraidx 61 mins. Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de> Only tested this with ktd data and `time` but it is consitently faster, awesome work.
Created attachment 179080 [details] [review] Bug 38408: Add parallel exporting in rebuild_zebra.pl The first part of the Zebra rebuild is the exporting. This part is made faster. The second part with zebraidx is not changed. A new commandline parameter -forks is added to the rebuild_zebra.pl script. A subroutine export_marc_records is added between index_records and export_marc_records_from_sth. The last routine has a new parameter, the sequence number of the export file. NOTE: This report does not touch koha-rebuild-zebra yet! This will be done on a follow-up. Test plan: Note that the number of forks/records below can be adjusted according to your server and database setup. [1] Reindex a subset of 100 records without forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -r -d /tmp/rebuild01 -k --length 100 Check if /tmp/rebuild01/biblio contains one export file for auth/bib. Verify that max. 100 auth and bib were indexed (check Auth search, Cataloguing) [2] Reindex an additional subset of 100 recs with forks (remove -r, add -forks): su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild02 -k --length 100 --offset 100 -forks 3 Check if /tmp/rebuild02/biblio contains 3 export files for auth/bib. Verify that max. 200 auth and bib were indexed (check Auth search, Cataloguing) [3] Run a full reindex with forks: su [YOUR_KOHA_USER] misc/migration_tools/rebuild_zebra.pl -a -b -d /tmp/rebuild03 -k -forks 3 Check both searches again. [4] Bonus: To get a feeling of improved speed, reindex a larger production db with and without using -forks. (Use something like above.) You may add -I to skip indexing in order to better compare both exports. Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl> Reindexed a prod db in 96 mins instead of 150 mins (3 forks, 4 cores). Main gain in biblio export; complete export took 35 mins, zebraidx 61 mins. Signed-off-by: Paul Derscheid <paul.derscheid@lmscloud.de> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Finally found a moment to revisit this.. very nice improvement. Passing QA
(In reply to Martin Renvoize (ashimema) from comment #13) > Finally found a moment to revisit this.. very nice improvement. > > Passing QA Great Martin. Thanks
Please document the new parameters in the script. --help gives nothing for the new parameters. Should it be --forks? Also release notes please!
Pushed for 25.05! Well done everyone, thank you!
Created attachment 179096 [details] [review] Bug 38408: (QA follow-up) Add --forks to usage statement Test plan: Run script with -h Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
(In reply to Katrin Fischer from comment #15) > Please document the new parameters in the script. > > --help gives nothing for the new parameters. > > Should it be --forks? > > Also release notes please! Ah we could also move to processes or so ? Just wait a bit
(In reply to Marcel de Rooy from comment #18) > (In reply to Katrin Fischer from comment #15) > > Please document the new parameters in the script. > > > > --help gives nothing for the new parameters. > > > > Should it be --forks? > > > > Also release notes please! > > Ah we could also move to processes or so ? Just wait a bit I'll wait for your "green light" before pushing the follow-ups, thanks Marcel.
Created attachment 179097 [details] [review] Bug 38408: (QA follow-up) Rename --forks, add usage Test plan: Run script with -h Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
(In reply to Katrin Fischer from comment #19) > (In reply to Marcel de Rooy from comment #18) > > (In reply to Katrin Fischer from comment #15) > > > Please document the new parameters in the script. > > > > > > --help gives nothing for the new parameters. > > > > > > Should it be --forks? > > > > > > Also release notes please! > > > > Ah we could also move to processes or so ? Just wait a bit > > I'll wait for your "green light" before pushing the follow-ups, thanks > Marcel. Back to green
Switching status so I see it for next RM session.