Bug 13706 - Deduping authorities script (dedup_authorities.pl)
Summary: Deduping authorities script (dedup_authorities.pl)
Status: ASSIGNED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: master
Hardware: All All
: P5 - low new feature with 31 votes (vote)
Assignee: Nick Clemens
QA Contact: Testopia
URL:
Keywords:
: 7419 (view as bug list)
Depends on: 13705 11700
Blocks:
  Show dependency treegraph
 
Reported: 2015-02-13 08:39 UTC by Julian Maurice
Modified: 2022-06-03 16:13 UTC (History)
12 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13706: New script dedup_authorities.pl (20.30 KB, patch)
2015-02-13 08:40 UTC, Julian Maurice
Details | Diff | Splinter Review
[Follow up] Bug 13706 - Fix QA failure (1.60 KB, patch)
2015-06-10 13:18 UTC, Alex Arnaud
Details | Diff | Splinter Review
[Follow up] Bug 13706 - Fix QA failure (1.60 KB, patch)
2015-06-10 13:18 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 13706: Update some code (6.33 KB, patch)
2021-06-14 18:20 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (11.70 KB, patch)
2021-06-16 18:34 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (11.89 KB, patch)
2021-09-08 15:59 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (12.53 KB, patch)
2021-11-03 13:59 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2015-02-13 08:39:32 UTC

    
Comment 1 Julian Maurice 2015-02-13 08:40:19 UTC
Created attachment 35885 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query, depending on --match parameter(s)
  2.2) run the query, retrieve the results
  2.3) for each result check if result is really a duplicate. check is
  customizable with option --check.
  2.4) among real duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) index updated biblios and authorities, and remove from index
 removed authorities

Use --help for more informations on options

Script is in misc/migration_tools/
Comment 2 Bernardo Gonzalez Kriegel 2015-05-02 01:40:22 UTC
On top of 13705.

1) No license
2) koha-qa complais

 FAIL   misc/migration_tools/dedup_authorities.pl
   FAIL   pod
                *** ERROR: Apparent command =cut not preceded by blank line  in file misc/migration_tools/dedup_authorities.pl
*** WARNING: =head2 without preceding higher level  in file misc/migration_tools/dedup_authorities.pl
   OK     forbidden patterns
   FAIL   critic
                Variable declared in conditional statement at line 429, column 5. Declare variables outside of the condition.
                Variable declared in conditional statement at line 435, column 5. Declare variables outside of the condition.
   FAIL   valid
                "my" variable $string masks earlier declaration in same scope 
                given is experimental 
                when is experimental
Comment 3 Alex Arnaud 2015-06-10 13:18:36 UTC Comment hidden (obsolete)
Comment 4 Alex Arnaud 2015-06-10 13:18:58 UTC
Created attachment 40069 [details] [review]
[Follow up] Bug 13706 - Fix QA failure
Comment 5 Mirko Tietgen 2016-01-28 13:27:52 UTC
No test plan available.
Comment 6 David Cook 2021-03-22 00:22:41 UTC
*** Bug 7419 has been marked as a duplicate of this bug. ***
Comment 7 Nick Clemens 2021-06-14 18:20:33 UTC
Created attachment 121920 [details] [review]
Bug 13706: Update some code

This updates a few things:
- ZebraIndex module does not exist
- merge calls ModBiblio, so the records are reindexed during merge
- DelAuthority used to remove records - no reindex needed after

Remaining issues:
- SimpleSearch doesn't work - only 'he' has an entry in ccl.properties
- abbreviations for indexes (he-main etc) do not exist anywhere
- There should be an option to use 'Match' and automatically generate the search term from the auth record (search_form in Headings.pm) based on the authtype
- The default options are unimarc centric
- this should be in module and testable
Comment 8 Nick Clemens 2021-06-16 18:34:34 UTC
Created attachment 122049 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements
Comment 9 Nick Clemens 2021-06-16 18:35:09 UTC
Stealing this one unless Alex or Julian object
Comment 10 Nick Clemens 2021-09-08 15:59:55 UTC
Created attachment 124659 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements
Comment 11 Nick Clemens 2021-11-03 13:59:32 UTC
Created attachment 127263 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements