If you read every word of the description of AutoCreateAuthorities in the system prefs page, and then read every word about it in the manual, even if you also read every word in the description and in the manual about LinkerModule, you'll have no idea that by turning on AutoCreateAuthorities without changing LinkerModule from its default value of Default, you've just given yourself infinite authority duplicates. The behavior of LinkerModule: Default is to link a field in a bib to an authority when there is one and only one matching authority record. Without AutoCreateAuthorities, that means if you have zero matches or two or more matches, the field is unlinked. Fine, no problem. With AutoCreateAuthorities on, the behavior is that if there are zero matches, an authority record is created, if there is one match, the field is linked to that existing authority record, if there are two matches a third duplicate authority is created, and if there are three matches, a fourth duplicate match is created, and if there are 2661 matches, a 2662nd duplicate match is created (that being how many authority records my system had for the genre Thrillers (Fiction) when I noticed this behavior). Two ways to not create infinite duplicates occur to me: Change what a LinkerModule returns, so it returns the match, whether it was a fuzzy match, and also whether more than one match was found, so that in the AutoCreateAuthorities case when more than one was found it can be left unlinked rather than creating another duplicate, or, Have C4/Linker/Default.pm check whether AutoCreateAuthorities is on, and if it is and $behavior isn't either first or last, just force it to first rather than creating infinite duplicates.
I like your first option better: teach AutoCreateAuthorities the difference between "there is no match" and "there's more than one possible match and therefore the linker didn't link anything."
Yeah, I was too focused on my half million duplicates that actually are duplicates resulting from this bug, and forgot that "duplicates" which only look like duplicates to the linker exist since it doesn't map indicators and possibly subfield 2 to 008/11 and 040$f in the authority to tell apart "650 _ 1 ‡aFantasy" a kids work of fantasy fiction and "650 _ 0 ‡aFantasy" an adult work about daydreaming. So forcing FirstMatch, which can only work correctly in certain limited circumstances, wouldn't be right.
Created attachment 103858 [details] [review] Bug 25189: Unit tests
Created attachment 103870 [details] [review] Bug 25189: Unit tests Adds new test for not adding authority if some already exist Also replaces use of 'SearchAuthorities' as it is Zebra specific
Created attachment 103871 [details] [review] Bug 25189: Don't create authority if results found Automatic authority creation assumes that if we don't match we need a new authority. Using the Default linker, however, we don't match if there exists more than one match. This leads to repeatedly generating authorities once there is a duplicate in the system We shoudl instead only create a new authority if there are no results To test: 1 - Set Linker Module to 'Default' 2 - Enable AutoCreateAuthorities and BiblioAddsAuthorities and CatalogModuleRelink and LinkerRelink 3 - Add two copies of a single authority via Z39 4 - Add a heading for that authority to a bib record 5 - Save the record and note a new authority is generated 6 - Repeat and see another is generated 7 - Apply patch 8 - Restart all the things 9 - Save the record again, no new authority created
Created attachment 103894 [details] [review] Bug 25189: Unit tests Adds new test for not adding authority if some already exist Also replaces use of 'SearchAuthorities' as it is Zebra specific Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Created attachment 103895 [details] [review] Bug 25189: Don't create authority if results found Automatic authority creation assumes that if we don't match we need a new authority. Using the Default linker, however, we don't match if there exists more than one match. This leads to repeatedly generating authorities once there is a duplicate in the system We shoudl instead only create a new authority if there are no results To test: 1 - Set Linker Module to 'Default' 2 - Enable AutoCreateAuthorities and BiblioAddsAuthorities and CatalogModuleRelink and LinkerRelink 3 - Add two copies of a single authority via Z39 4 - Add a heading for that authority to a bib record 5 - Save the record and note a new authority is generated 6 - Repeat and see another is generated 7 - Apply patch 8 - Restart all the things 9 - Save the record again, no new authority created Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Created attachment 106073 [details] [review] Bug 25189: Unit tests Adds new test for not adding authority if some already exist Also replaces use of 'SearchAuthorities' as it is Zebra specific Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 106074 [details] [review] Bug 25189: Don't create authority if results found Automatic authority creation assumes that if we don't match we need a new authority. Using the Default linker, however, we don't match if there exists more than one match. This leads to repeatedly generating authorities once there is a duplicate in the system We shoudl instead only create a new authority if there are no results To test: 1 - Set Linker Module to 'Default' 2 - Enable AutoCreateAuthorities and BiblioAddsAuthorities and CatalogModuleRelink and LinkerRelink 3 - Add two copies of a single authority via Z39 4 - Add a heading for that authority to a bib record 5 - Save the record and note a new authority is generated 6 - Repeat and see another is generated 7 - Apply patch 8 - Restart all the things 9 - Save the record again, no new authority created Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Change in behaviour makes sense and works as expected, tests pass, qa script passes. Passing QA
Pushed to master for 20.11, thanks to everybody involved!
backported to 20.05.x for 20.05.02
Backported to 19.05.x branch for 19.05.17
I think that this is a beneficial change, as I have a library generating thousands of duplicates due to getting too many search results (due to a Zebra ICU bug it seems). However, I've been thinking it might be good if we can prevent cataloguing editor saves if we can't link headings to authorities, or provide a summary of the linking. The latter would be better for batch imports. I mention this here only because interested parties CCed here might have thoughts...
Victor and me are working on backport to 19.11.x
(In reply to Victor Grousset/tuxayo from comment #13) > Backported to 19.05.x branch for 19.05.17 oops, old message sneaked in. Backported to 19.11.x branch for 19.11.13
Question for Phil Ringnalda: Phil, do you know if you're using CHR or ICU indexing?
(In reply to David Cook from comment #14) > > However, I've been thinking it might be good if we can prevent cataloguing > editor saves if we can't link headings to authorities, or provide a summary > of the linking. The latter would be better for batch imports. > > I mention this here only because interested parties CCed here might have > thoughts... I love this idea, David! It would probably need to be two enhancements. One for individual record saves that would prompt for authorities that would be created or can't be linked if the auto-create permission is turned off. The second would be a summary report of linking, failed links, and newly created authorities for batch imports. Our copy cataloging workflow is a one-at-a-time import via Z39.50, so something that would prompt before the record is saved would probably work better for us. If prompted, the cataloger could go back through the record to manually link any fields that came back with too many search results to link. Please share when you have bug numbers for this and thank you for the idea!
Nick added a See Also to Bug 11299 which adds a button for doing the linking while cataloguing along with a summary at that time. Not quite what I had in mind, but it seems like it would be very useful.