To recreate: 1 - have LinkerModule set to First Match 2 - have a bib with its author linked to an authority 3 - duplicate that authority, confirm that an authority search for your author brings up two identical authorities, one linked to your bib 4 - merge the two authority records, telling Koha to keep the newer of the two (the one not linked to your bib) 5 - search authorities for your author again, confirm you see just one authority and it's not linked to any bibs 6 - reload your bib, see the 100$9 contains the number for the authority you deleted in the merge 7 - edit your bib to correctly link it to the remaining authority 8 - set LinkerModule to Default 9 - repeat steps 3 and 4 10 - see your remaining authority is linked to your bib, the bib has the correct authority number I'm not able to replicate this in master. I'm not sure what changed from 21.05.
I managed to recreate this on master. Here are my system preferences: - AutoCreateAuthorities = don't generate - RequireChoosingExistingAuthority = don't require - AutoLinkBiblios = Do - CatalogModuleRelink = Do - LinkerKeepStale = Don't - LinkerModule = first match - LinkerRelink = Do I use Elasticsearch with default mappings. Here is my complete test plan. 1. Create an authority record 1.1. Go to Authorities 1.2. Click on "New authority" and choose an authority type (I chose Corporate name)* 1.3. Fill in the mandatory fields (000, 003, 005, 008, 040$c) 1.4. In tab 1, click on the label of the main heading (110 for Corporate name) 1.5. Fill in subfield $a 1.6. Copy content of subfield $a 1.7. Click on "Save" 1.8. Note the authority record number 2. Duplicate the authority record 2.1. Click on "Edit" and choose "Edit as new (duplicate)" 2.2. Click on "Save" 2.3. Click on "No: Save as new authority" 2.4. Note the authority record number (should be the number from step 1.8 +1) 3. Link one of the authority records to a bibliographic record 3.1 In another tab, do a catalog search 3.2. Click on "Edit record" under one of the search results 3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7) 3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles) 3.4. Paste the text previously copied in subfield $a 3.5. Click on "Link authorities automatically" at the top of the page 3.6. Note the authority record number in subfield $9 --> The linked authority record should be the first authority record (step 1.8) 3.7. Click on "Save" 4. Merge the two authority records, keep the one that is not linked to the bibliographic record 4.1. Go back to the tab with the authorities 4.2. Click on "Actions" and choose "Merge" for both records 4.3. Select the record number that is NOT linked to the bibliographic record (should be the one already selected) 4.4. Click on "Next" 4.5. Click on "Merge" 5. Check the bibliographic record 5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5) 5.2. Hover the mouse cursor over the name or term, without clicking 5.3. Note the record number that appears at the bottom of the screen --> The record number was not changed I also tried to run merge_authorities, without success. *I have found the linker doesn't seem to work in 6XX fields, so for this test plan, choose something that can go in 7XX (personal name, corporate name, or meeting name)
Created attachment 152061 [details] [review] Bug 29522 - Bib record not correctly updated when merging identical authorities with LinkerModule set to First Match To test: Set system preferences AutoCreateAuthorities = don't generate RequireChoosingExistingAuthority = don't require AutoLinkBiblios = Do CatalogModuleRelink = Do LinkerKeepStale = Don't LinkerModule = first match LinkerRelink = Do 1. Create an authority record 1.1. Go to Authorities 1.2. Click on "New authority" and choose an authority type (I chose Corporate name)* 1.3. Fill in the mandatory fields (000, 003, 005, 008, 040) 1.4. In tab 1, click on the label of the main heading (110 for Corporate name) 1.5. Fill in subfield 1.6. Copy content of subfield 1.7. Click on "Save" 1.8. Note the authority record number 2. Duplicate the authority record 2.1. Click on "Edit" and choose "Edit as new (duplicate)" 2.2. Click on "Save" 2.3. Click on "No: Save as new authority" 2.4. Note the authority record number (should be the number from step 1.8 +1) 3. Link one of the authority records to a bibliographic record 3.1 In another tab, do a catalog search 3.2. Click on "Edit record" under one of the search results 3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7) 3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles) 3.4. Paste the text previously copied in subfield 3.5. Click on "Link authorities automatically" at the top of the page 3.6. Note the authority record number in subfield --> The linked authority record should be the first authority record (step 1.8) 3.7. Click on "Save" 4. Merge the two authority records, keep the one that is not linked to the bibliographic record 4.1. Go back to the tab with the authorities 4.1. Go to Authorities 4.2. Search for the aurhorities previous created 4.2. Click on "Actions" and choose "Merge" for both records 4.3. Select the record number that is NOT linked to the bibliographic record 4.4. Click on "Next" 4.5. Click on "Merge" 5. Check the bibliographic record 5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5) 5.2. Hover the mouse cursor over the name or term, without clicking 5.3. Note the record number that appears at the bottom of the screen --> The record number was not changed 6. Apply the patch 6.1. Run prove t/AuthoritiesMarc_MARC21.t 6.2. Repeat Previous steps 1, 2, 3, 4, 5 --> The record number is change
Created attachment 152062 [details] [review] Bug 29522 - Adding unit Test
You aren't actually checking whether they are duplicates, only whether the first is a duplicate-or-subset of the second, ignoring order and ignoring repeated subfields. $aCastles$zFrance is not a duplicate of $aCastles$zFrance$zProvence, and $aFoo$xBar$yQuux is not a duplicate of $aFoo$yQuux$xBar (and if you are getting subfield in scalar context so asking for subfield z always gets the first z even if there's more than one, there might be one more non-duplicate case). But isAuthorityDuplicated() is the wrong question to ask, because that's not the problem case. The problem is wouldLinkerPreferMARCfrom. Consider the all too common case of having authid 1 with 13473 bibs linked, authid 1000 with 1 bib linked, and authid 1001 with 1 bib linked, where all three are exact duplicates. Currently, merging 1001 to 1000 with FirstMatch will correctly result in all 13475 bibs linked to authid 1. Your patch would instead link two bibs to authid 1000, until either authid 1000 or each of the bib records was saved again, when they would relink to authid 1. You also have to consider the problem behind Caroline's "I have found the linker doesn't seem to work in 6XX fields" comment, which would have been from the problem that bug 30280 didn't yet have enough followups to work. Now that it (probably) does, if you have three authorities with 110 $aPotato, answering whether or not wouldLinkerPreferMARCfrom depends not just on how many duplicate authorities there are but also on what the value of LinkerConsiderThesaurus is and what thesaurus each of the duplicates has. I don't think you can solve this problem without asking the linker itself whether it would prefer MARCfrom. (Or, by giving authority merges a way to say "update the index, do it now, and tell me when it's done" which is probably too much responsiveness to even dream of.)
Created attachment 155278 [details] [review] Bug 29522: [alternate] Skip relinking bibliographic records when merging authorities If autolinking is enabled, there are scenarios where the authorities merge process updates the record, then sends it to ModBiblio which then AutoLinks the record, undoing the changes in the record (as the 'from' authority is not removed until the process is complete) This patch simply disables the autolinking process when merging authorities To test: Set system preferences AutoCreateAuthorities = don't generate RequireChoosingExistingAuthority = don't require AutoLinkBiblios = Do CatalogModuleRelink = Do LinkerKeepStale = Don't LinkerModule = first match LinkerRelink = Do 1. Create an authority record 1.1. Go to Authorities 1.2. Click on "New authority" and choose an authority type (I chose Corporate name)* 1.3. Fill in the mandatory fields (000, 003, 005, 008, 040) 1.4. In tab 1, click on the label of the main heading (110 for Corporate name) 1.5. Fill in subfield 1.6. Copy content of subfield 1.7. Click on "Save" 1.8. Note the authority record number 2. Duplicate the authority record 2.1. Click on "Edit" and choose "Edit as new (duplicate)" 2.2. Click on "Save" 2.3. Click on "No: Save as new authority" 2.4. Note the authority record number (should be the number from step 1.8 +1) 3. Link one of the authority records to a bibliographic record 3.1 In another tab, do a catalog search 3.2. Click on "Edit record" under one of the search results 3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7) 3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles) 3.4. Paste the text previously copied in subfield 3.5. Click on "Link authorities automatically" at the top of the page 3.6. Note the authority record number in subfield --> The linked authority record should be the first authority record (step 1.8) 3.7. Click on "Save" 4. Merge the two authority records, keep the one that is not linked to the bibliographic record 4.1. Go back to the tab with the authorities 4.1. Go to Authorities 4.2. Search for the aurhorities previous created 4.2. Click on "Actions" and choose "Merge" for both records 4.3. Select the record number that is NOT linked to the bibliographic record 4.4. Click on "Next" 4.5. Click on "Merge" 5. Check the bibliographic record 5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5) 5.2. Hover the mouse cursor over the name or term, without clicking 5.3. Note the record number that appears at the bottom of the screen --> The record number was not changed 6. Apply the patch 6.1. Run prove t/AuthoritiesMarc_MARC21.t 6.2. Repeat Previous steps 1, 2, 3, 4, 5 --> The record number is change
I supplied a patch, going a different route. Please test and let me know if you think this makes sense. It will require unit tests, however, there are currently no tests in this module so I wanted to see if this patch works as expected before delving into the tests. There are three scenarios where we call merge: 1 - When merging from an old authority to a new 2 - When updating an authority 3 - When deleting an authority In each case, I believe running an autolink is not needed: 1 - Here we have specified the merge, so we have chosen the record we wish to be linked to 2 - We are changing a heading and want to change the linked records - we don't want to link them elsewhere 3 - When deleting we will remove the headings, so there is nothing left to relink
There is a fourth scenario: when someone is intentionally updating an authority in such a way that autolink will realize it should link to something else, in order to avoid an actual old->new merge. authorities/merge.pl is optimized for merges where you are merging authid 10 to authid 1, but you are keeping the 100 tag from 10, or you are keeping the 008+040$f from 10 to give 1 a different $2 in the linked bib records, so that there's a reason to update the headings in bibs linked to the kept authority record. Whether or not the merge will make any difference to linked bibs, it calls ModAuthority on 1, triggering a 1->1 merge by doing so, and then merges 10->1. That makes merging Thrillers (Ficition) with 1 linked bib to Thrillers (Fiction) with 13534 linked bibs a bad idea (unless you have MergeLimit over 13534), since deferred merges aren't particularly robust. Even below MergeLimit, it's a waste of time and electrons to update every linked bib when you haven't made any changes that will update anything. So all day every day, I change "Patterson, James" to "Patterson, James, 1947-" in an authority record, expecting the attached bib to realize it should relink to the existing authority. My extensive use of something isn't necessarily a reason not to break it, but removing relinks for updating a single authority would be a huge workflow frustration for me.
(In reply to Phil Ringnalda from comment #7) > There is a fourth scenario: when someone is intentionally updating an > authority in such a way that autolink will realize it should link to > something else, in order to avoid an actual old->new merge. > > authorities/merge.pl is optimized for merges where you are merging authid 10 > to authid 1, but you are keeping the 100 tag from 10, or you are keeping the > 008+040$f from 10 to give 1 a different $2 in the linked bib records, so > that there's a reason to update the headings in bibs linked to the kept > authority record. Whether or not the merge will make any difference to > linked bibs, it calls ModAuthority on 1, triggering a 1->1 merge by doing > so, and then merges 10->1. > > That makes merging Thrillers (Ficition) with 1 linked bib to Thrillers > (Fiction) with 13534 linked bibs a bad idea (unless you have MergeLimit over > 13534), since deferred merges aren't particularly robust. Even below > MergeLimit, it's a waste of time and electrons to update every linked bib > when you haven't made any changes that will update anything. So all day > every day, I change "Patterson, James" to "Patterson, James, 1947-" in an > authority record, expecting the attached bib to realize it should relink to > the existing authority. My extensive use of something isn't necessarily a > reason not to break it, but removing relinks for updating a single authority > would be a huge workflow frustration for me. It sounds like you are describing related bugs that you have created a workaround to avoid: 1 - Deferred merges aren't reliable? Could you file a bug with more details on what the problem is with deferred merges? 2 - Linked biblios should not be updated when the authorized heading, including indicators and thesaurus values, have not changed I don't think it is a general expectation that merging or updating authorities would end up linking the biblio headings to a different authority.
I started avoiding deferred merges in a database impacted by bug 25189, so I don't think I can write steps to reproduce that anyone will follow. They'd be things like "create an authority record, link 2000 bibs to it, create 3000 duplicate authority records and link 1 bib to each, create another authority record and link 4000 bibs to it, create 6000 duplicate authority records and link 1 bib to each, with 1500 of those also linked to one of the duplicates of the other record, and 1500 also linked to the very first authority record, then merge all the things, triggering another run of the cronjob each time your fingers get tired to stack up multiple runs." And the solution to that thing that nobody would actually do? Bug 34739.
Created attachment 159837 [details] [review] Bug 29522: [alternate] Skip relinking bibliographic records when merging authorities If autolinking is enabled, there are scenarios where the authorities merge process updates the record, then sends it to ModBiblio which then AutoLinks the record, undoing the changes in the record (as the 'from' authority is not removed until the process is complete) This patch simply disables the autolinking process when merging authorities To test: Set system preferences AutoCreateAuthorities = don't generate RequireChoosingExistingAuthority = don't require AutoLinkBiblios = Do CatalogModuleRelink = Do LinkerKeepStale = Don't LinkerModule = first match LinkerRelink = Do 1. Create an authority record 1.1. Go to Authorities 1.2. Click on "New authority" and choose an authority type (I chose Corporate name)* 1.3. Fill in the mandatory fields (000, 003, 005, 008, 040) 1.4. In tab 1, click on the label of the main heading (110 for Corporate name) 1.5. Fill in subfield 1.6. Copy content of subfield 1.7. Click on "Save" 1.8. Note the authority record number 2. Duplicate the authority record 2.1. Click on "Edit" and choose "Edit as new (duplicate)" 2.2. Click on "Save" 2.3. Click on "No: Save as new authority" 2.4. Note the authority record number (should be the number from step 1.8 +1) 3. Link one of the authority records to a bibliographic record 3.1 In another tab, do a catalog search 3.2. Click on "Edit record" under one of the search results 3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7) 3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles) 3.4. Paste the text previously copied in subfield 3.5. Click on "Link authorities automatically" at the top of the page 3.6. Note the authority record number in subfield --> The linked authority record should be the first authority record (step 1.8) 3.7. Click on "Save" 4. Merge the two authority records, keep the one that is not linked to the bibliographic record 4.1. Go back to the tab with the authorities 4.1. Go to Authorities 4.2. Search for the aurhorities previous created 4.2. Click on "Actions" and choose "Merge" for both records 4.3. Select the record number that is NOT linked to the bibliographic record 4.4. Click on "Next" 4.5. Click on "Merge" 5. Check the bibliographic record 5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5) 5.2. Hover the mouse cursor over the name or term, without clicking 5.3. Note the record number that appears at the bottom of the screen --> The record number was not changed 6. Apply the patch 6.1. Run prove t/AuthoritiesMarc_MARC21.t 6.2. Repeat Previous steps 1, 2, 3, 4, 5 --> The record number is change Signed-off-by: Barbara Petritsch <barbara.petritsch@wienmuseum.at>
Very nicely documented - Thank You
Any chance of a unit test for this Nick?
Created attachment 161525 [details] [review] Bug 29522: [alternate] Skip relinking bibliographic records when merging authorities If autolinking is enabled, there are scenarios where the authorities merge process updates the record, then sends it to ModBiblio which then AutoLinks the record, undoing the changes in the record (as the 'from' authority is not removed until the process is complete) This patch simply disables the autolinking process when merging authorities To test: Set system preferences AutoCreateAuthorities = don't generate RequireChoosingExistingAuthority = don't require AutoLinkBiblios = Do CatalogModuleRelink = Do LinkerKeepStale = Don't LinkerModule = first match LinkerRelink = Do 1. Create an authority record 1.1. Go to Authorities 1.2. Click on "New authority" and choose an authority type (I chose Corporate name)* 1.3. Fill in the mandatory fields (000, 003, 005, 008, 040) 1.4. In tab 1, click on the label of the main heading (110 for Corporate name) 1.5. Fill in subfield 1.6. Copy content of subfield 1.7. Click on "Save" 1.8. Note the authority record number 2. Duplicate the authority record 2.1. Click on "Edit" and choose "Edit as new (duplicate)" 2.2. Click on "Save" 2.3. Click on "No: Save as new authority" 2.4. Note the authority record number (should be the number from step 1.8 +1) 3. Link one of the authority records to a bibliographic record 3.1 In another tab, do a catalog search 3.2. Click on "Edit record" under one of the search results 3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7) 3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles) 3.4. Paste the text previously copied in subfield 3.5. Click on "Link authorities automatically" at the top of the page 3.6. Note the authority record number in subfield --> The linked authority record should be the first authority record (step 1.8) 3.7. Click on "Save" 4. Merge the two authority records, keep the one that is not linked to the bibliographic record 4.1. Go back to the tab with the authorities 4.1. Go to Authorities 4.2. Search for the aurhorities previous created 4.2. Click on "Actions" and choose "Merge" for both records 4.3. Select the record number that is NOT linked to the bibliographic record 4.4. Click on "Next" 4.5. Click on "Merge" 5. Check the bibliographic record 5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5) 5.2. Hover the mouse cursor over the name or term, without clicking 5.3. Note the record number that appears at the bottom of the screen --> The record number was not changed 6. Apply the patch 6.1. Run prove t/AuthoritiesMarc_MARC21.t 6.2. Repeat Previous steps 1, 2, 3, 4, 5 --> The record number is change Signed-off-by: Barbara Petritsch <barbara.petritsch@wienmuseum.at>
Created attachment 161526 [details] [review] Bug 29522: Unit test
+ $biblio_module->mock( + 'ModBiblio', + sub { + my ( undef, undef, undef, $params ) = @_; + warn "Auto link disabled" if $params->{disable_autolink}; + } + ); I was hoping for some nice test showing that it was not autolinked. And without the code change showing that it was before.. This is a very simplistic approach :) No blocker for me, but this test seems a bit unneeded..
Created attachment 163187 [details] [review] Bug 29522: [alternate] Skip relinking bibliographic records when merging authorities If autolinking is enabled, there are scenarios where the authorities merge process updates the record, then sends it to ModBiblio which then AutoLinks the record, undoing the changes in the record (as the 'from' authority is not removed until the process is complete) This patch simply disables the autolinking process when merging authorities To test: Set system preferences AutoCreateAuthorities = don't generate RequireChoosingExistingAuthority = don't require AutoLinkBiblios = Do CatalogModuleRelink = Do LinkerKeepStale = Don't LinkerModule = first match LinkerRelink = Do 1. Create an authority record 1.1. Go to Authorities 1.2. Click on "New authority" and choose an authority type (I chose Corporate name)* 1.3. Fill in the mandatory fields (000, 003, 005, 008, 040) 1.4. In tab 1, click on the label of the main heading (110 for Corporate name) 1.5. Fill in subfield 1.6. Copy content of subfield 1.7. Click on "Save" 1.8. Note the authority record number 2. Duplicate the authority record 2.1. Click on "Edit" and choose "Edit as new (duplicate)" 2.2. Click on "Save" 2.3. Click on "No: Save as new authority" 2.4. Note the authority record number (should be the number from step 1.8 +1) 3. Link one of the authority records to a bibliographic record 3.1 In another tab, do a catalog search 3.2. Click on "Edit record" under one of the search results 3.3. Go to the relevant tab (for corporate name, I used field 710 in tab 7) 3.3.a. If there are no empty fields for that tag, click the "Repeat this tag" button (two rectangles) 3.4. Paste the text previously copied in subfield 3.5. Click on "Link authorities automatically" at the top of the page 3.6. Note the authority record number in subfield --> The linked authority record should be the first authority record (step 1.8) 3.7. Click on "Save" 4. Merge the two authority records, keep the one that is not linked to the bibliographic record 4.1. Go back to the tab with the authorities 4.1. Go to Authorities 4.2. Search for the aurhorities previous created 4.2. Click on "Actions" and choose "Merge" for both records 4.3. Select the record number that is NOT linked to the bibliographic record 4.4. Click on "Next" 4.5. Click on "Merge" 5. Check the bibliographic record 5.1. Go back to the tab with the bibliographic record and refresh the page (Ctrl+F5) 5.2. Hover the mouse cursor over the name or term, without clicking 5.3. Note the record number that appears at the bottom of the screen --> The record number was not changed 6. Apply the patch 6.1. Run prove t/AuthoritiesMarc_MARC21.t 6.2. Repeat Previous steps 1, 2, 3, 4, 5 --> The record number is change Signed-off-by: Barbara Petritsch <barbara.petritsch@wienmuseum.at> Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Created attachment 163188 [details] [review] Bug 29522: Unit test Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Pushed for 24.05! Well done everyone, thank you!
Pushed to 23.11.x for 23.11.04
Backported to 23.05.x for upcoming 23.05.10