Bug 25189 - AutoCreateAuthorities can repeatedly generate authority records when using Default linker
Summary: AutoCreateAuthorities can repeatedly generate authority records when using De...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on:
Blocks: 28676
  Show dependency treegraph
 
Reported: 2020-04-19 03:33 UTC by Phil Ringnalda
Modified: 2022-06-06 20:24 UTC (History)
10 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.02, 19.11.13


Attachments
Bug 25189: Unit tests (1.94 KB, patch)
2020-04-28 13:30 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25189: Unit tests (2.81 KB, patch)
2020-04-28 15:39 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25189: Don't create authority if results found (3.16 KB, patch)
2020-04-28 15:39 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25189: Unit tests (2.88 KB, patch)
2020-04-29 01:12 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 25189: Don't create authority if results found (3.22 KB, patch)
2020-04-29 01:12 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 25189: Unit tests (2.94 KB, patch)
2020-06-19 11:50 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 25189: Don't create authority if results found (3.29 KB, patch)
2020-06-19 11:50 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Phil Ringnalda 2020-04-19 03:33:08 UTC
If you read every word of the description of AutoCreateAuthorities in the system prefs page, and then read every word about it in the manual, even if you also read every word in the description and in the manual about LinkerModule, you'll have no idea that by turning on AutoCreateAuthorities without changing LinkerModule from its default value of Default, you've just given yourself infinite authority duplicates.

The behavior of LinkerModule: Default is to link a field in a bib to an authority when there is one and only one matching authority record. Without AutoCreateAuthorities, that means if you have zero matches or two or more matches, the field is unlinked. Fine, no problem.

With AutoCreateAuthorities on, the behavior is that if there are zero matches, an authority record is created, if there is one match, the field is linked to that existing authority record, if there are two matches a third duplicate authority is created, and if there are three matches, a fourth duplicate match is created, and if there are 2661 matches, a 2662nd duplicate match is created (that being how many authority records my system had for the genre Thrillers (Fiction) when I noticed this behavior).

Two ways to not create infinite duplicates occur to me:

Change what a LinkerModule returns, so it returns the match, whether it was a fuzzy match, and also whether more than one match was found, so that in the AutoCreateAuthorities case when more than one was found it can be left unlinked rather than creating another duplicate, or,

Have C4/Linker/Default.pm check whether AutoCreateAuthorities is on, and if it is and $behavior isn't either first or last, just force it to first rather than creating infinite duplicates.
Comment 1 Andrew Fuerste-Henry 2020-04-20 12:35:59 UTC
I like your first option better: teach AutoCreateAuthorities the difference between "there is no match" and "there's more than one possible match and therefore the linker didn't link anything."
Comment 2 Phil Ringnalda 2020-04-20 15:44:23 UTC
Yeah, I was too focused on my half million duplicates that actually are duplicates resulting from this bug, and forgot that "duplicates" which only look like duplicates to the linker exist since it doesn't map indicators and possibly subfield 2 to 008/11 and 040$f in the authority to tell apart "650 _ 1 ‡aFantasy" a kids work of fantasy fiction and "650 _ 0 ‡aFantasy" an adult work about daydreaming. So forcing FirstMatch, which can only work correctly in certain limited circumstances, wouldn't be right.
Comment 3 Nick Clemens 2020-04-28 13:30:12 UTC
Created attachment 103858 [details] [review]
Bug 25189: Unit tests
Comment 4 Nick Clemens 2020-04-28 15:39:49 UTC
Created attachment 103870 [details] [review]
Bug 25189: Unit tests

Adds new test for not adding authority if some already exist

Also replaces use of 'SearchAuthorities' as it is Zebra specific
Comment 5 Nick Clemens 2020-04-28 15:39:52 UTC
Created attachment 103871 [details] [review]
Bug 25189: Don't create authority if results found

Automatic authority creation assumes that if we don't match we need a new authority.

Using the Default linker, however, we don't match if there exists more than one match.
This leads to repeatedly generating authorities once there is a duplicate in the system

We shoudl instead only create a new authority if there are no results

To test:
1 - Set Linker Module to 'Default'
2 - Enable  AutoCreateAuthorities  and  BiblioAddsAuthorities and  CatalogModuleRelink and LinkerRelink
3 - Add two copies of a single authority via Z39
4 - Add a heading for that authority to a bib record
5 - Save the record and note a new authority is generated
6 - Repeat and see another is generated
7 - Apply patch
8 - Restart all the things
9 - Save the record again, no new authority created
Comment 6 Andrew Fuerste-Henry 2020-04-29 01:12:14 UTC
Created attachment 103894 [details] [review]
Bug 25189: Unit tests

Adds new test for not adding authority if some already exist

Also replaces use of 'SearchAuthorities' as it is Zebra specific

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 7 Andrew Fuerste-Henry 2020-04-29 01:12:17 UTC
Created attachment 103895 [details] [review]
Bug 25189: Don't create authority if results found

Automatic authority creation assumes that if we don't match we need a new authority.

Using the Default linker, however, we don't match if there exists more than one match.
This leads to repeatedly generating authorities once there is a duplicate in the system

We shoudl instead only create a new authority if there are no results

To test:
1 - Set Linker Module to 'Default'
2 - Enable  AutoCreateAuthorities  and  BiblioAddsAuthorities and  CatalogModuleRelink and LinkerRelink
3 - Add two copies of a single authority via Z39
4 - Add a heading for that authority to a bib record
5 - Save the record and note a new authority is generated
6 - Repeat and see another is generated
7 - Apply patch
8 - Restart all the things
9 - Save the record again, no new authority created

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 8 Martin Renvoize 2020-06-19 11:50:35 UTC
Created attachment 106073 [details] [review]
Bug 25189: Unit tests

Adds new test for not adding authority if some already exist

Also replaces use of 'SearchAuthorities' as it is Zebra specific

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize 2020-06-19 11:50:38 UTC
Created attachment 106074 [details] [review]
Bug 25189: Don't create authority if results found

Automatic authority creation assumes that if we don't match we need a new authority.

Using the Default linker, however, we don't match if there exists more than one match.
This leads to repeatedly generating authorities once there is a duplicate in the system

We shoudl instead only create a new authority if there are no results

To test:
1 - Set Linker Module to 'Default'
2 - Enable  AutoCreateAuthorities  and  BiblioAddsAuthorities and  CatalogModuleRelink and LinkerRelink
3 - Add two copies of a single authority via Z39
4 - Add a heading for that authority to a bib record
5 - Save the record and note a new authority is generated
6 - Repeat and see another is generated
7 - Apply patch
8 - Restart all the things
9 - Save the record again, no new authority created

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize 2020-06-19 11:51:31 UTC
Change in behaviour makes sense and works as expected, tests pass, qa script passes.

Passing QA
Comment 11 Jonathan Druart 2020-06-24 13:32:05 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 12 Lucas Gass 2020-07-10 15:56:27 UTC
backported to 20.05.x for 20.05.02
Comment 13 Victor Grousset/tuxayo 2020-12-18 22:17:31 UTC
Backported to 19.05.x branch for 19.05.17
Comment 14 David Cook 2020-12-20 23:29:06 UTC
I think that this is a beneficial change, as I have a library generating thousands of duplicates due to getting too many search results (due to a Zebra ICU bug it seems).

However, I've been thinking it might be good if we can prevent cataloguing editor saves if we can't link headings to authorities, or provide a summary of the linking. The latter would be better for batch imports. 

I mention this here only because interested parties CCed here might have thoughts...
Comment 15 Fridolin Somers 2020-12-21 10:13:02 UTC
Victor and me are working on backport to 19.11.x
Comment 16 Victor Grousset/tuxayo 2020-12-21 12:39:32 UTC
(In reply to Victor Grousset/tuxayo from comment #13)
> Backported to 19.05.x branch for 19.05.17

oops, old message sneaked in.

Backported to 19.11.x branch for 19.11.13
Comment 17 David Cook 2020-12-21 22:44:56 UTC
Question for Phil Ringnalda:

Phil, do you know if you're using CHR or ICU indexing?
Comment 18 Myka Kennedy Stephens 2020-12-24 16:06:01 UTC
(In reply to David Cook from comment #14)
> 
> However, I've been thinking it might be good if we can prevent cataloguing
> editor saves if we can't link headings to authorities, or provide a summary
> of the linking. The latter would be better for batch imports. 
> 
> I mention this here only because interested parties CCed here might have
> thoughts...

I love this idea, David! It would probably need to be two enhancements. One for individual record saves that would prompt for authorities that would be created or can't be linked if the auto-create permission is turned off. The second would be a summary report of linking, failed links, and newly created authorities for batch imports. 

Our copy cataloging workflow is a one-at-a-time import via Z39.50, so something that would prompt before the record is saved would probably work better for us. If prompted, the cataloger could go back through the record to manually link any fields that came back with too many search results to link. 

Please share when you have bug numbers for this and thank you for the idea!
Comment 19 David Cook 2021-01-03 23:29:55 UTC
Nick added a See Also to Bug 11299 which adds a button for doing the linking while cataloguing along with a summary at that time. Not quite what I had in mind, but it seems like it would be very useful.