Bug 13706 - Deduping authorities script (dedup_authorities.pl)
Summary: Deduping authorities script (dedup_authorities.pl)
Status: Passed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: Main
Hardware: All All
: P5 - low new feature with 33 votes (vote)
Assignee: Nick Clemens
QA Contact: Tomás Cohen Arazi
URL:
Keywords:
: 7419 (view as bug list)
Depends on: 13705 11700
Blocks:
  Show dependency treegraph
 
Reported: 2015-02-13 08:39 UTC by Julian Maurice
Modified: 2024-04-09 06:28 UTC (History)
16 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This new command line script (misc/maintenance/dedup_authorities.pl) is a tool for helping to automatically remove duplicate authority records. Features: - Choose a method(s) to select which authority record to keep when duplicates are found. Methods include: . date: keep the most recent authority (based on 005 field) . used: keep the most used authority . ppn: PPN (UNIMARC only), keep the authority with a ppn (when some authorities don't have one, based on 009 field) - Use a SQL WHERE statement to limit the authority records checked for deduplication - Check only specified authority types - Increase the level of detail shown using the --verbose option - Changes are only made when the --confirm option is used Examples: - Methods - for the authorities that have a PPN, keep the most recent, and if two (or more) have the same date in 005, keep the most used: --method ppn --method date --method used - SQL WHERE statement - only look at records with an auth_id less than 5,000: --where="authid < 5000" - Limit deduplication to specific authority types: --authtypecode PERSO_NAME See the script help for the options available and usage examples (misc/maintenance/dedup_authorities.pl --help).
Version(s) released in:


Attachments
Bug 13706: New script dedup_authorities.pl (20.30 KB, patch)
2015-02-13 08:40 UTC, Julian Maurice
Details | Diff | Splinter Review
[Follow up] Bug 13706 - Fix QA failure (1.60 KB, patch)
2015-06-10 13:18 UTC, Alex Arnaud
Details | Diff | Splinter Review
[Follow up] Bug 13706 - Fix QA failure (1.60 KB, patch)
2015-06-10 13:18 UTC, Alex Arnaud
Details | Diff | Splinter Review
Bug 13706: Update some code (6.33 KB, patch)
2021-06-14 18:20 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (11.70 KB, patch)
2021-06-16 18:34 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (11.89 KB, patch)
2021-09-08 15:59 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (12.53 KB, patch)
2021-11-03 13:59 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (12.34 KB, patch)
2023-10-26 16:28 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (12.38 KB, patch)
2023-10-26 23:52 UTC, David Nind
Details | Diff | Splinter Review
Bug 13706: New script dedup_authorities.pl (12.44 KB, patch)
2024-03-11 13:26 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review
Bug 13706: Minor tidy/spelling issues (1.35 KB, patch)
2024-03-11 13:26 UTC, Tomás Cohen Arazi
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Julian Maurice 2015-02-13 08:39:32 UTC

    
Comment 1 Julian Maurice 2015-02-13 08:40:19 UTC
Created attachment 35885 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query, depending on --match parameter(s)
  2.2) run the query, retrieve the results
  2.3) for each result check if result is really a duplicate. check is
  customizable with option --check.
  2.4) among real duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) index updated biblios and authorities, and remove from index
 removed authorities

Use --help for more informations on options

Script is in misc/migration_tools/
Comment 2 Bernardo Gonzalez Kriegel 2015-05-02 01:40:22 UTC
On top of 13705.

1) No license
2) koha-qa complais

 FAIL   misc/migration_tools/dedup_authorities.pl
   FAIL   pod
                *** ERROR: Apparent command =cut not preceded by blank line  in file misc/migration_tools/dedup_authorities.pl
*** WARNING: =head2 without preceding higher level  in file misc/migration_tools/dedup_authorities.pl
   OK     forbidden patterns
   FAIL   critic
                Variable declared in conditional statement at line 429, column 5. Declare variables outside of the condition.
                Variable declared in conditional statement at line 435, column 5. Declare variables outside of the condition.
   FAIL   valid
                "my" variable $string masks earlier declaration in same scope 
                given is experimental 
                when is experimental
Comment 3 Alex Arnaud 2015-06-10 13:18:36 UTC Comment hidden (obsolete)
Comment 4 Alex Arnaud 2015-06-10 13:18:58 UTC
Created attachment 40069 [details] [review]
[Follow up] Bug 13706 - Fix QA failure
Comment 5 Mirko Tietgen 2016-01-28 13:27:52 UTC
No test plan available.
Comment 6 David Cook 2021-03-22 00:22:41 UTC
*** Bug 7419 has been marked as a duplicate of this bug. ***
Comment 7 Nick Clemens 2021-06-14 18:20:33 UTC
Created attachment 121920 [details] [review]
Bug 13706: Update some code

This updates a few things:
- ZebraIndex module does not exist
- merge calls ModBiblio, so the records are reindexed during merge
- DelAuthority used to remove records - no reindex needed after

Remaining issues:
- SimpleSearch doesn't work - only 'he' has an entry in ccl.properties
- abbreviations for indexes (he-main etc) do not exist anywhere
- There should be an option to use 'Match' and automatically generate the search term from the auth record (search_form in Headings.pm) based on the authtype
- The default options are unimarc centric
- this should be in module and testable
Comment 8 Nick Clemens 2021-06-16 18:34:34 UTC
Created attachment 122049 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements
Comment 9 Nick Clemens 2021-06-16 18:35:09 UTC
Stealing this one unless Alex or Julian object
Comment 10 Nick Clemens 2021-09-08 15:59:55 UTC
Created attachment 124659 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements
Comment 11 Nick Clemens 2021-11-03 13:59:32 UTC
Created attachment 127263 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements
Comment 12 Katrin Fischer 2023-07-28 21:38:13 UTC
Is this a forgotten one? Still sounds interesting!
Comment 13 Nick Clemens 2023-07-31 17:44:18 UTC
We use this in production as-is - if you like it Katrin, feel free to sign-off - I would like it all to be covered by tests but don't know when I will have a chance to work on that
Comment 14 Katrin Fischer 2023-07-31 19:11:03 UTC
I came here looking through old bugs - a new past time of mine. I am setting to NSO then.
Comment 15 David Nind 2023-09-24 06:01:35 UTC
I had a go at testing, but I got some errors when running the script:

- with the help option:

misc/maintenance/dedup_authorities.pl --help
Undefined subroutine &main::pod2usage called at misc/maintenance/dedup_authorities.pl line 99.

- duplicated an authority record, the attempted to run using the date option:

misc/maintenance/dedup_authorities.pl -v -c -m date
Fetching authtypecodes...
The method Koha::Authority::Types->authtypecode is not covered by tests!

Trace begun at /kohadevbox/koha/Koha/Objects.pm line 572
Koha::Objects::AUTOLOAD('Koha::Authority::Types=HASH(0x564621d97918)') called at misc/maintenance/dedup_authorities.pl line 132
Comment 16 Nick Clemens 2023-10-26 16:28:33 UTC
Created attachment 157935 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements
Comment 17 David Nind 2023-10-26 23:52:44 UTC
Created attachment 157965 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

Signed-off-by: David Nind <david@davidnind.com>
Comment 18 David Nind 2023-10-27 00:00:07 UTC
It seems to work with my limited testing - if this isn't sufficient, please change the status.

The only thing I found confusing was the message shown when run "Deleted XX authorities" - I'm assuming there is some logic here that I don't quite get.

Basic testing notes (using KTD):

1. Go to Authorities
2. Duplicate an existing personal name authority that is used in a record
3. Note the original and duplicate authority number
4. Run the script: misc/maintenance/dedup_authorities.pl -v -a PERSO_NAME -m date -c
5. Check the authorities in the staff interface - one is deleted and one is kept

I did some other testing, notes as follows.

FYI 
===

Number of terms for authority types - from the script and search results in the staff interface:

Chronological (CHRON_TERM): Script: 0 ; Search result: 0
Corporate Name (CORPO_NAME): Script: 88 ; Search result: 88
Genre/Form Term (GENRE/FORM): Script: 49 ; Search result: 49
Geographic Name (GEOGR_NAME): Script: 142 ; Search result: 142
Meeting Name (MEETI_NAME): Script: 3 ; Search result: 3
Personal Name (PERSO_NAME): Script: 650 ; Search result: 650
Topical Term (TOPIC_TERM): Script: 663 ; Search result: 663
Uniform Title (UNIF_TITLE): Script: 111 ; Search result: 111

Default/No authority type selected: Script: 1706 ; Search result: 1706

Script result - showing authority records by type:

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -m date
RUNNING IN TEST MODE, NO CHANGES WILL BE MADE
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype '' 
Fetching authorities for ''... 0 authorities found
End of deduping for authtype ''
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'CHRON_TERM' 
Fetching authorities for 'CHRON_TERM'... 0 authorities found
End of deduping for authtype 'CHRON_TERM'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'CORPO_NAME' 
Fetching authorities for 'CORPO_NAME'... 88 authorities found
End of deduping for authtype 'CORPO_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'GENRE/FORM' 
Fetching authorities for 'GENRE/FORM'... 49 authorities found
End of deduping for authtype 'GENRE/FORM'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'GEOGR_NAME' 
Fetching authorities for 'GEOGR_NAME'... 142 authorities found
Progression for authtype 'GEOGR_NAME': 100/142 (70.42%)
End of deduping for authtype 'GEOGR_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'MEETI_NAME' 
Fetching authorities for 'MEETI_NAME'... 3 authorities found
End of deduping for authtype 'MEETI_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'PERSO_NAME' 
Fetching authorities for 'PERSO_NAME'... 650 authorities found
    Malformed authority record, no heading at misc/maintenance/dedup_authorities.pl line 172.
Progression for authtype 'PERSO_NAME': 100/650 (15.38%)
Progression for authtype 'PERSO_NAME': 200/650 (30.77%)
Progression for authtype 'PERSO_NAME': 300/650 (46.15%)
Progression for authtype 'PERSO_NAME': 400/650 (61.54%)
Progression for authtype 'PERSO_NAME': 500/650 (76.92%)
    Malformed authority record, blank heading at misc/maintenance/dedup_authorities.pl line 176.
End of deduping for authtype 'PERSO_NAME'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'TOPIC_TERM' 
Fetching authorities for 'TOPIC_TERM'... 663 authorities found
Progression for authtype 'TOPIC_TERM': 100/663 (15.08%)
Progression for authtype 'TOPIC_TERM': 200/663 (30.17%)
Progression for authtype 'TOPIC_TERM': 300/663 (45.25%)
Progression for authtype 'TOPIC_TERM': 400/663 (60.33%)
Progression for authtype 'TOPIC_TERM': 500/663 (75.41%)
End of deduping for authtype 'TOPIC_TERM'
Updated 0 biblios
Deleted 0 authorities
Deduping authtype 'UNIF_TITLE' 
Fetching authorities for 'UNIF_TITLE'... 111 authorities found
End of deduping for authtype 'UNIF_TITLE'
Updated 0 biblios
Deleted 0 authorities
No biblios to update


Testing using Genre/Form Term
=============================

Summary
~~~~~~~

Total before script run: 49 terms

Manually went through results in staff interface to identify duplicates:

Commedy films. 982 (deleted), 1586 (kept)
Feature films. 625 (deleted), 650 (deleted), 654 (deleted), 822 (deleted), 984 (kept), 987 (deleted)
Fiction films. 823 (deleted), 985 (kept), 988 (deleted)
Foreign films. 626 (kept), 988 (deleted)
Historical fiction. 1018 (kept), 1019 (deleted)
Video recordings for the hearing impaired. 986 (kept), 989 (deleted)
Summary = 17 terms, should only be 6 = 11 should be deleted

Expected number of terms to be deleted: 11 (result would be 38 terms left)

Results from running the script - shows 14 deleted (3 shown as deleted twice): 987, 822, 625, 625, 650, 654, 655, 988, 823, 823, 982, 982, 989, 1019

End result is the same, not sure why showing as it does.

Search results after in the staff interface: 38 
Script results after: 38

Dummy run
~~~~~~~~~

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -a GENRE/FORM -m date 
RUNNING IN TEST MODE, NO CHANGES WILL BE MADE
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype 'GENRE/FORM' 
Fetching authorities for 'GENRE/FORM'... 49 authorities found
End of deduping for authtype 'GENRE/FORM'
Updated 0 biblios
Deleted 0 authorities
No biblios to update

Actual run
~~~~~~~~~~

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -a GENRE/FORM -m date -c
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype 'GENRE/FORM' 
Fetching authorities for 'GENRE/FORM'... 49 authorities found
    Updated 0 biblios
    Deleting 987
    Updated 0 biblios
    Deleting 822
    Updated 0 biblios
    Deleting 625
    Updated 0 biblios
    Deleting 625
    Updated 0 biblios
    Deleting 650
    Updated 0 biblios
    Deleting 654
    Updated 0 biblios
    Deleting 655
    Updated 0 biblios
    Deleting 988
    Updated 0 biblios
    Deleting 823
    Updated 0 biblios
    Deleting 823
    Updated 0 biblios
    Deleting 982
    Updated 0 biblios
    Deleting 982
    Updated 0 biblios
    Deleting 989
    Updated 0 biblios
    Deleting 1019
End of deduping for authtype 'GENRE/FORM'
Updated 0 biblios
Deleted 14 authorities
No biblios to update

After
~~~~~~

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -a GENRE/FORM -m date
RUNNING IN TEST MODE, NO CHANGES WILL BE MADE
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype 'GENRE/FORM' 
Fetching authorities for 'GENRE/FORM'... 38 authorities found
End of deduping for authtype 'GENRE/FORM'
Updated 0 biblios
Deleted 0 authorities
No biblios to update


Testing using Meeting Name authorities
======================================

Summary
~~~~~~~

1. Duplicated the Beagle Expedition (1831-1836) authority (155)
2. Added the duplicated authority to another record
3. Result expected - one authority deleted, one kept - records using authority updated with the kept authority

Total before script run: 4 terms

Manually went through results in staff interface to identify duplicates:

Beagle Expedition (1831-1836): 155 (deleted), 1708 (kept)
Summary = 2 terms, should only be 1 = 1 should be deleted

Expected number of terms to be deleted: 1 (result would be 3 terms left)

Results from running the script - shows 2 deleted (1 shown as deleted twice): 155, 155

End result is the same, not sure why showing as it does.

Search results after in the staff interface: 3
Script results after: 3

Dummy run
~~~~~~~~~

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -a MEETI_NAME -m date
RUNNING IN TEST MODE, NO CHANGES WILL BE MADE
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype 'MEETI_NAME' 
Fetching authorities for 'MEETI_NAME'... 4 authorities found
End of deduping for authtype 'MEETI_NAME'
Updated 0 biblios
Deleted 0 authorities
No biblios to update

Actual run
~~~~~~~~~~

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -a MEETI_NAME -m date -c
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype 'MEETI_NAME' 
Fetching authorities for 'MEETI_NAME'... 4 authorities found
    Updated 1 biblios
    Deleting 155
    Updated 0 biblios
    Deleting 155
End of deduping for authtype 'MEETI_NAME'
Updated 1 biblios
Deleted 2 authorities
No biblios to update

After
~~~~~

kohadev-koha@kohadevbox:koha(bz13706)$ misc/maintenance/dedup_authorities.pl -v -a MEETI_NAME -m date
RUNNING IN TEST MODE, NO CHANGES WILL BE MADE
Fetching authtypecodes...
Fetching authtypecodes done.
Deduping authtype 'MEETI_NAME' 
Fetching authorities for 'MEETI_NAME'... 3 authorities found
End of deduping for authtype 'MEETI_NAME'
Updated 0 biblios
Deleted 0 authorities
No biblios to update
Comment 19 Tomás Cohen Arazi 2024-03-11 13:26:11 UTC
Created attachment 163030 [details] [review]
Bug 13706: New script dedup_authorities.pl

This script allow to deduplicate authorities automatically.

Script is in misc/maintenance/

It works this way:
 1) authorities are fetched from the database. You can limit fetched
 results by authtypecode, or directly by specifying WHERE clause
 2) for each authority:
  2.1) build a Zebra query using the 'search_form' for the heading
  2.2) run the query, retrieve the results
  2.3) among duplicates, choose the one we want to keep (use
  --choose-method option).
  2.5) use C4::Authorities::merge to merge authorities
 3) delete the merged authorities

Use --help for more informations on options

To be done:
1 - Move to module and cover with tests
2 - Add option to only merge unused authorities
3 - Expand 'ppn' option to be 'control-number' option and allow specifying field
4 - More?

1 & 2 I will attempt - 3 & 4 may be future enhancements

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 20 Tomás Cohen Arazi 2024-03-11 13:26:15 UTC
Created attachment 163031 [details] [review]
Bug 13706: Minor tidy/spelling issues

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 21 Tomás Cohen Arazi 2024-03-11 13:27:26 UTC
This is a nice new feature, self-contained on a new script.

I have some doubts about the output format, but I think we want to listen from real-life users and enhance this later with good feedback from users.

Nice job, Nick!!!
Comment 22 David Cook 2024-04-09 06:28:41 UTC
Ohhh I kept meaning to look at this. 

I wrote an authority de-duplicator years ago, and I'm planning on reviewing it soon. 

Maybe I can replace my local one with this one. I'll have to take a look...