Bug 25189 - AutoCreateAuthorities can repeatedly generate authority records when using Default linker
Summary: AutoCreateAuthorities can repeatedly generate authority records when using De...
Status: Pushed to stable
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2020-04-19 03:33 UTC by Phil Ringnalda
Modified: 2020-07-10 15:56 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:
20.11.00, 20.05.02


Attachments
Bug 25189: Unit tests (1.94 KB, patch)
2020-04-28 13:30 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25189: Unit tests (2.81 KB, patch)
2020-04-28 15:39 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25189: Don't create authority if results found (3.16 KB, patch)
2020-04-28 15:39 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 25189: Unit tests (2.88 KB, patch)
2020-04-29 01:12 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 25189: Don't create authority if results found (3.22 KB, patch)
2020-04-29 01:12 UTC, Andrew Fuerste-Henry
Details | Diff | Splinter Review
Bug 25189: Unit tests (2.94 KB, patch)
2020-06-19 11:50 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 25189: Don't create authority if results found (3.29 KB, patch)
2020-06-19 11:50 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Phil Ringnalda 2020-04-19 03:33:08 UTC
If you read every word of the description of AutoCreateAuthorities in the system prefs page, and then read every word about it in the manual, even if you also read every word in the description and in the manual about LinkerModule, you'll have no idea that by turning on AutoCreateAuthorities without changing LinkerModule from its default value of Default, you've just given yourself infinite authority duplicates.

The behavior of LinkerModule: Default is to link a field in a bib to an authority when there is one and only one matching authority record. Without AutoCreateAuthorities, that means if you have zero matches or two or more matches, the field is unlinked. Fine, no problem.

With AutoCreateAuthorities on, the behavior is that if there are zero matches, an authority record is created, if there is one match, the field is linked to that existing authority record, if there are two matches a third duplicate authority is created, and if there are three matches, a fourth duplicate match is created, and if there are 2661 matches, a 2662nd duplicate match is created (that being how many authority records my system had for the genre Thrillers (Fiction) when I noticed this behavior).

Two ways to not create infinite duplicates occur to me:

Change what a LinkerModule returns, so it returns the match, whether it was a fuzzy match, and also whether more than one match was found, so that in the AutoCreateAuthorities case when more than one was found it can be left unlinked rather than creating another duplicate, or,

Have C4/Linker/Default.pm check whether AutoCreateAuthorities is on, and if it is and $behavior isn't either first or last, just force it to first rather than creating infinite duplicates.
Comment 1 Andrew Fuerste-Henry 2020-04-20 12:35:59 UTC
I like your first option better: teach AutoCreateAuthorities the difference between "there is no match" and "there's more than one possible match and therefore the linker didn't link anything."
Comment 2 Phil Ringnalda 2020-04-20 15:44:23 UTC
Yeah, I was too focused on my half million duplicates that actually are duplicates resulting from this bug, and forgot that "duplicates" which only look like duplicates to the linker exist since it doesn't map indicators and possibly subfield 2 to 008/11 and 040$f in the authority to tell apart "650 _ 1 ‡aFantasy" a kids work of fantasy fiction and "650 _ 0 ‡aFantasy" an adult work about daydreaming. So forcing FirstMatch, which can only work correctly in certain limited circumstances, wouldn't be right.
Comment 3 Nick Clemens 2020-04-28 13:30:12 UTC
Created attachment 103858 [details] [review]
Bug 25189: Unit tests
Comment 4 Nick Clemens 2020-04-28 15:39:49 UTC
Created attachment 103870 [details] [review]
Bug 25189: Unit tests

Adds new test for not adding authority if some already exist

Also replaces use of 'SearchAuthorities' as it is Zebra specific
Comment 5 Nick Clemens 2020-04-28 15:39:52 UTC
Created attachment 103871 [details] [review]
Bug 25189: Don't create authority if results found

Automatic authority creation assumes that if we don't match we need a new authority.

Using the Default linker, however, we don't match if there exists more than one match.
This leads to repeatedly generating authorities once there is a duplicate in the system

We shoudl instead only create a new authority if there are no results

To test:
1 - Set Linker Module to 'Default'
2 - Enable  AutoCreateAuthorities  and  BiblioAddsAuthorities and  CatalogModuleRelink and LinkerRelink
3 - Add two copies of a single authority via Z39
4 - Add a heading for that authority to a bib record
5 - Save the record and note a new authority is generated
6 - Repeat and see another is generated
7 - Apply patch
8 - Restart all the things
9 - Save the record again, no new authority created
Comment 6 Andrew Fuerste-Henry 2020-04-29 01:12:14 UTC
Created attachment 103894 [details] [review]
Bug 25189: Unit tests

Adds new test for not adding authority if some already exist

Also replaces use of 'SearchAuthorities' as it is Zebra specific

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 7 Andrew Fuerste-Henry 2020-04-29 01:12:17 UTC
Created attachment 103895 [details] [review]
Bug 25189: Don't create authority if results found

Automatic authority creation assumes that if we don't match we need a new authority.

Using the Default linker, however, we don't match if there exists more than one match.
This leads to repeatedly generating authorities once there is a duplicate in the system

We shoudl instead only create a new authority if there are no results

To test:
1 - Set Linker Module to 'Default'
2 - Enable  AutoCreateAuthorities  and  BiblioAddsAuthorities and  CatalogModuleRelink and LinkerRelink
3 - Add two copies of a single authority via Z39
4 - Add a heading for that authority to a bib record
5 - Save the record and note a new authority is generated
6 - Repeat and see another is generated
7 - Apply patch
8 - Restart all the things
9 - Save the record again, no new authority created

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 8 Martin Renvoize 2020-06-19 11:50:35 UTC
Created attachment 106073 [details] [review]
Bug 25189: Unit tests

Adds new test for not adding authority if some already exist

Also replaces use of 'SearchAuthorities' as it is Zebra specific

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize 2020-06-19 11:50:38 UTC
Created attachment 106074 [details] [review]
Bug 25189: Don't create authority if results found

Automatic authority creation assumes that if we don't match we need a new authority.

Using the Default linker, however, we don't match if there exists more than one match.
This leads to repeatedly generating authorities once there is a duplicate in the system

We shoudl instead only create a new authority if there are no results

To test:
1 - Set Linker Module to 'Default'
2 - Enable  AutoCreateAuthorities  and  BiblioAddsAuthorities and  CatalogModuleRelink and LinkerRelink
3 - Add two copies of a single authority via Z39
4 - Add a heading for that authority to a bib record
5 - Save the record and note a new authority is generated
6 - Repeat and see another is generated
7 - Apply patch
8 - Restart all the things
9 - Save the record again, no new authority created

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize 2020-06-19 11:51:31 UTC
Change in behaviour makes sense and works as expected, tests pass, qa script passes.

Passing QA
Comment 11 Jonathan Druart 2020-06-24 13:32:05 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 12 Lucas Gass 2020-07-10 15:56:27 UTC
backported to 20.05.x for 20.05.02