When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field.
I've conducted the following tests with a modified version of the script that accepts the MARC field to operate on as an argument (my Koha DB has approximately 22k biblio records and 7k authority records): Command line used Time (MM:SS) Comments ----------------------------------------------- ------------ -------- link_bibs_to_authorities.pl -v -l --test 05:49 1st run link_bibs_to_authorities.pl -v -l --test 06:07 2nd run link_bibs_to_authorities.pl -v -l --test 06:40 3rd run Command line used Time (MM:SS) Comments ----------------------------------------------- ------------ -------- link_bibs_to_authorities.pl -v -l --test -g=700 03:02 1st run link_bibs_to_authorities.pl -v -l --test -g=710 00:58 1st run link_bibs_to_authorities.pl -v -l --test -g=710 00:58 2nd run
Created attachment 111395 [details] [review] Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. This patch provides that enhancement. Test plan: 1) Run the script as follows (preferably in a test DB): time link_bibs_to_authorities.pl -v -l --test ... and notice how long it takes to complete. 2) Apply the patch. 3) Run the script again, this time providing the MARC field to work on: time link_bibs_to_authorities.pl -v -l --test -g=700 ... and, again, notice how long it takes to complete. The 2nd run should run faster than the 1st one.
You must have been reading my mind last week, because I was just about to start work on something similar.
Comment on attachment 111395 [details] [review] Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl Review of attachment 111395 [details] [review]: ----------------------------------------------------------------- ::: misc/link_bibs_to_authorities.pl @@ +200,4 @@ > } > > my $frameworkcode = GetFrameworkCode($biblionumber); > + my $allowrelink = C4::Context->preference("CatalogModuleRelink") || ''; From a performance (and reliability) perspective, this lookup should be done outside the function and before the while loop. I can submit a follow-up patch for that.
Actually, looks like we had something different in mind, but that's OK. I'll still work on this anyway.
Created attachment 111461 [details] [review] Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. This patch provides that enhancement. Test plan: 1) Run the script as follows (preferably in a test DB): time link_bibs_to_authorities.pl -v -l --test ... and notice how long it takes to complete. 2) Apply the patch. 3) Run the script again, this time providing the MARC field to work on: time link_bibs_to_authorities.pl -v -l --test -g=700 ... and, again, notice how long it takes to complete. The 2nd run should run faster than the 1st one. Signed-off-by: David Cook <dcook@prosentient.com.au>
Created attachment 111462 [details] [review] Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without
Andreas, if you test and sign off my patch, you can move this to "Signed Off".
Created attachment 111463 [details] [review] Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without
(In reply to David Cook from comment #8) > Andreas, if you test and sign off my patch, you can move this to "Signed > Off". Thank you for your interest in this bug report, David. I've set up a koha-testing-docker development environment and will now go through your test plan.
Created attachment 111829 [details] [review] Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without Signed-off-by: Andreas Roussos <a.roussos@dataly.gr>
Created attachment 111868 [details] [review] Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl When running link_bibs_to_authorities.pl, it could be useful to have the ability to specify which MARC field the script should operate on. For example, when you have a catalog where most links are in place, and you don't want the script to process every MARC field but rather limit it to a specific MARC field (e.g. 700) because you've observed that there are many missing links for that particular MARC field. This patch provides that enhancement. Test plan: 1) Run the script as follows (preferably in a test DB): time link_bibs_to_authorities.pl -v -l --test ... and notice how long it takes to complete. 2) Apply the patch. 3) Run the script again, this time providing the MARC field to work on: time link_bibs_to_authorities.pl -v -l --test -g=700 ... and, again, notice how long it takes to complete. The 2nd run should run faster than the 1st one. Signed-off-by: David Cook <dcook@prosentient.com.au> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 111869 [details] [review] Bug 26641: Optimize CatalogModuleRelink lookup This patch moves the CatalogModuleRelink lookup outside the loop to improve performance (especially for large databases), and includes a tester friendly test plan. Test plan: 0. Using koha-testing-docker 1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev 2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked 3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev 4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked 5. Note output appear to all be names (rather than subjects or other authorities) 6. Note also that the 2nd run with -g 700 is almost 2x faster than without Signed-off-by: Andreas Roussos <a.roussos@dataly.gr> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Love the teamwork here! All seems to work as expected and I'm not seeing any regressions. QA Scripts happy.. Passing QA
why "g"?
Pushed to master for 20.11, thanks to everybody involved!
enhancement will not be backported to 20.05.x
It's a shame that we didn't make this a repeatable option...
(In reply to Jonathan Druart from comment #15) > why "g"? Because "t" was already taken (for the -t/--test option). Apologies for necro posting ;-)