Bug 26641 - link_bibs_to_authorities.pl: Add the ability to specify the MARC field to operate on
Summary: link_bibs_to_authorities.pl: Add the ability to specify the MARC field to ope...
Status: Pushed to master
Alias: None
Product: Koha
Classification: Unclassified
Component: Command-line Utilities (show other bugs)
Version: master
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Andreas Roussos
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-10-09 11:29 UTC by Andreas Roussos
Modified: 2020-12-21 01:45 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Text to go in the release notes:
Version(s) released in:
20.11.00


Attachments
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl (4.87 KB, patch)
2020-10-09 11:56 UTC, Andreas Roussos
Details | Diff | Splinter Review
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl (4.92 KB, patch)
2020-10-11 23:53 UTC, David Cook
Details | Diff | Splinter Review
Bug 26641: Optimize CatalogModuleRelink lookup (3.14 KB, patch)
2020-10-11 23:53 UTC, David Cook
Details | Diff | Splinter Review
Bug 26641: Optimize CatalogModuleRelink lookup (3.29 KB, patch)
2020-10-11 23:58 UTC, David Cook
Details | Diff | Splinter Review
Bug 26641: Optimize CatalogModuleRelink lookup (3.34 KB, patch)
2020-10-16 07:15 UTC, Andreas Roussos
Details | Diff | Splinter Review
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl (4.99 KB, patch)
2020-10-16 12:35 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 26641: Optimize CatalogModuleRelink lookup (3.41 KB, patch)
2020-10-16 12:35 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Andreas Roussos 2020-10-09 11:29:57 UTC
When running link_bibs_to_authorities.pl, it could be useful to have
the ability to specify which MARC field the script should operate on.

For example, when you have a catalog where most links are in place,
and you don't want the script to process every MARC field but rather
limit it to a specific MARC field (e.g. 700) because you've observed
that there are many missing links for that particular MARC field.
Comment 1 Andreas Roussos 2020-10-09 11:55:56 UTC
I've conducted the following tests with a modified version of the script
that accepts the MARC field to operate on as an argument (my Koha DB has approximately 22k biblio records and 7k authority records):

Command line used                                Time (MM:SS)  Comments
-----------------------------------------------  ------------  --------
link_bibs_to_authorities.pl -v -l --test         05:49         1st run
link_bibs_to_authorities.pl -v -l --test         06:07         2nd run
link_bibs_to_authorities.pl -v -l --test         06:40         3rd run

Command line used                                Time (MM:SS)  Comments
-----------------------------------------------  ------------  --------
link_bibs_to_authorities.pl -v -l --test -g=700  03:02         1st run
link_bibs_to_authorities.pl -v -l --test -g=710  00:58         1st run
link_bibs_to_authorities.pl -v -l --test -g=710  00:58         2nd run
Comment 2 Andreas Roussos 2020-10-09 11:56:53 UTC
Created attachment 111395 [details] [review]
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl

When running link_bibs_to_authorities.pl, it could be useful to have
the ability to specify which MARC field the script should operate on.

For example, when you have a catalog where most links are in place,
and you don't want the script to process every MARC field but rather
limit it to a specific MARC field (e.g. 700) because you've observed
that there are many missing links for that particular MARC field.

This patch provides that enhancement.

Test plan:
1) Run the script as follows (preferably in a test DB):

   time link_bibs_to_authorities.pl -v -l --test

   ... and notice how long it takes to complete.

2) Apply the patch.

3) Run the script again, this time providing the MARC field to work on:

   time link_bibs_to_authorities.pl -v -l --test -g=700

   ... and, again, notice how long it takes to complete.

   The 2nd run should run faster than the 1st one.
Comment 3 David Cook 2020-10-11 23:06:48 UTC
You must have been reading my mind last week, because I was just about to start work on something similar.
Comment 4 David Cook 2020-10-11 23:10:13 UTC
Comment on attachment 111395 [details] [review]
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl

Review of attachment 111395 [details] [review]:
-----------------------------------------------------------------

::: misc/link_bibs_to_authorities.pl
@@ +200,4 @@
>      }
>  
>      my $frameworkcode = GetFrameworkCode($biblionumber);
> +    my $allowrelink = C4::Context->preference("CatalogModuleRelink") || '';

From a performance (and reliability) perspective, this lookup should be done outside the function and before the while loop.

I can submit a follow-up patch for that.
Comment 5 David Cook 2020-10-11 23:47:31 UTC
Actually, looks like we had something different in mind, but that's OK. I'll still work on this anyway.
Comment 6 David Cook 2020-10-11 23:53:45 UTC
Created attachment 111461 [details] [review]
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl

When running link_bibs_to_authorities.pl, it could be useful to have
the ability to specify which MARC field the script should operate on.

For example, when you have a catalog where most links are in place,
and you don't want the script to process every MARC field but rather
limit it to a specific MARC field (e.g. 700) because you've observed
that there are many missing links for that particular MARC field.

This patch provides that enhancement.

Test plan:
1) Run the script as follows (preferably in a test DB):

   time link_bibs_to_authorities.pl -v -l --test

   ... and notice how long it takes to complete.

2) Apply the patch.

3) Run the script again, this time providing the MARC field to work on:

   time link_bibs_to_authorities.pl -v -l --test -g=700

   ... and, again, notice how long it takes to complete.

   The 2nd run should run faster than the 1st one.

Signed-off-by: David Cook <dcook@prosentient.com.au>
Comment 7 David Cook 2020-10-11 23:53:49 UTC
Created attachment 111462 [details] [review]
Bug 26641: Optimize CatalogModuleRelink lookup

This patch moves the CatalogModuleRelink lookup outside the loop to
improve performance (especially for large databases), and includes a
tester friendly test plan.

Test plan:
0. Using koha-testing-docker
1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev
2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked
3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev
4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked
5. Note output appear to all be names (rather than subjects or other authorities)
6. Note also that the 2nd run with -g 700 is almost 2x faster than without
Comment 8 David Cook 2020-10-11 23:54:21 UTC
Andreas, if you test and sign off my patch, you can move this to "Signed Off".
Comment 9 David Cook 2020-10-11 23:58:22 UTC
Created attachment 111463 [details] [review]
Bug 26641: Optimize CatalogModuleRelink lookup

This patch moves the CatalogModuleRelink lookup outside the loop to
improve performance (especially for large databases), and includes a
tester friendly test plan.

Test plan:
0. Using koha-testing-docker
1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev
2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked
3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev
4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked
5. Note output appear to all be names (rather than subjects or other authorities)
6. Note also that the 2nd run with -g 700 is almost 2x faster than without
Comment 10 Andreas Roussos 2020-10-16 06:40:29 UTC
(In reply to David Cook from comment #8)
> Andreas, if you test and sign off my patch, you can move this to "Signed
> Off".
Thank you for your interest in this bug report, David.
I've set up a koha-testing-docker development environment and will
now go through your test plan.
Comment 11 Andreas Roussos 2020-10-16 07:15:48 UTC
Created attachment 111829 [details] [review]
Bug 26641: Optimize CatalogModuleRelink lookup

This patch moves the CatalogModuleRelink lookup outside the loop to
improve performance (especially for large databases), and includes a
tester friendly test plan.

Test plan:
0. Using koha-testing-docker
1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev
2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked
3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev
4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked
5. Note output appear to all be names (rather than subjects or other authorities)
6. Note also that the 2nd run with -g 700 is almost 2x faster than without

Signed-off-by: Andreas Roussos <a.roussos@dataly.gr>
Comment 12 Martin Renvoize 2020-10-16 12:35:10 UTC
Created attachment 111868 [details] [review]
Bug 26641: pass the MARC field as an argument to link_bibs_to_authorities.pl

When running link_bibs_to_authorities.pl, it could be useful to have
the ability to specify which MARC field the script should operate on.

For example, when you have a catalog where most links are in place,
and you don't want the script to process every MARC field but rather
limit it to a specific MARC field (e.g. 700) because you've observed
that there are many missing links for that particular MARC field.

This patch provides that enhancement.

Test plan:
1) Run the script as follows (preferably in a test DB):

   time link_bibs_to_authorities.pl -v -l --test

   ... and notice how long it takes to complete.

2) Apply the patch.

3) Run the script again, this time providing the MARC field to work on:

   time link_bibs_to_authorities.pl -v -l --test -g=700

   ... and, again, notice how long it takes to complete.

   The 2nd run should run faster than the 1st one.

Signed-off-by: David Cook <dcook@prosentient.com.au>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 13 Martin Renvoize 2020-10-16 12:35:16 UTC
Created attachment 111869 [details] [review]
Bug 26641: Optimize CatalogModuleRelink lookup

This patch moves the CatalogModuleRelink lookup outside the loop to
improve performance (especially for large databases), and includes a
tester friendly test plan.

Test plan:
0. Using koha-testing-docker
1. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test" kohadev
2. Note 436 bibs checked, 1315 headings linked, 676 headings unlinked
3. koha-shell -c "./misc/link_bibs_to_authorities.pl -l --test -g 700" kohadev
4. Note 436 bibs checked, 248 headings linked, 111 headings unlinked
5. Note output appear to all be names (rather than subjects or other authorities)
6. Note also that the 2nd run with -g 700 is almost 2x faster than without

Signed-off-by: Andreas Roussos <a.roussos@dataly.gr>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 14 Martin Renvoize 2020-10-16 12:35:56 UTC
Love the teamwork here!

All seems to work as expected and I'm not seeing any regressions.

QA Scripts happy..

Passing QA
Comment 15 Jonathan Druart 2020-11-04 10:22:18 UTC
why "g"?
Comment 16 Jonathan Druart 2020-11-04 12:03:56 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 17 Lucas Gass 2020-11-16 21:24:45 UTC
enhancement will not be backported to 20.05.x
Comment 18 David Cook 2020-12-21 01:45:16 UTC
It's a shame that we didn't make this a repeatable option...