If an authority record is modified in any way, we trigger a merge to update all biblios using the authority. This also happens if we merge a duplciate authority into one that is in use, all biblios already attached to the correct authority are updated. If the heading or controlled indicators or thesaurus have changed, this makes sense, but if not, we are triggering many biblio modifications and reindexes that are not needed. We also need to consider the case where a See-From is is added and IncludeSeeFromInSearches is enabled
There is a development about adding URIs from 024 in auth to bib record when merging. A Biblibre report, forgot the number. This would be an exception.
Bug 22972
Created attachment 175665 [details] [review] Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow
Created attachment 175666 [details] [review] Bug 34739: Unit tests BTW, there is a tiny adjustment to one of the previous subtests to fit to the modified merge behaviour.
*** Bug 38742 has been marked as a duplicate of this bug. ***
Created attachment 175667 [details] [review] Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Created attachment 175668 [details] [review] Bug 34739: Unit tests BTW, there is a tiny adjustment to one of the previous subtests to fit to the modified merge behaviour. Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
# Subtest: ModBiblio calls from merge 1..4 not ok 1 - real modification of bibliographic record, ModBiblio called # Failed test 'real modification of bibliographic record, ModBiblio called' # at t/db_dependent/Authority/Merge.t line 612. # didn't find a warning # expected to find warning: ModBiblio called not ok 2 - no real modification of bibliographic record, index_records called # Failed test 'no real modification of bibliographic record, index_records called' # at t/db_dependent/Authority/Merge.t line 626. # didn't find a warning # expected to find warning: index_records called ok 3 - no real modification of bibliographic record, index_records not called not ok 4 - Only one bibliographic record updated while merging two authorities # Failed test 'Only one bibliographic record updated while merging two authorities' # at t/db_dependent/Authority/Merge.t line 655. # got: '0' # expected: '1' # Looks like you failed 3 tests of 4. not ok 13 - ModBiblio calls from merge
(In reply to Marcel de Rooy from comment #8) > not ok 13 - ModBiblio calls from merge This is somehow strange... Could you provide some more detailed information or repeat the test, please? I have tried anew on two separate fresh ktd installation -- everything looks clear: kohadev-koha@kohadevbox:koha(bug_34739)$ qa testing 2 commit(s) (applied to da08fc5 '1a Bug 31270: (follow-up) Fix descrip') Processing files before patches |========================>| 2 / 2 (100.00%) Processing files after patches |========================>| 2 / 2 (100.00%) OK C4/AuthoritiesMarc.pm OK t/db_dependent/Authority/Merge.t Processing additional checks OK! Running tests (1) * Proving /kohadevbox/koha/t/db_dependent/Authority/Merge.t OK!
Running tests (1) * Proving /usr/share/koha/t/db_dependent/Authority/Merge.t KO! # Failed test 'real modification of bibliographic record, ModBiblio called' # at /usr/share/koha/t/db_dependent/Authority/Merge.t line 612. # didn't find a warning # expected to find warning: ModBiblio called # Failed test 'no real modification of bibliographic record, index_records called' # at /usr/share/koha/t/db_dependent/Authority/Merge.t line 626. # didn't find a warning # expected to find warning: index_records called # Failed test 'Only one bibliographic record updated while merging two authorities' # at /usr/share/koha/t/db_dependent/Authority/Merge.t line 655. # got: '0' # expected: '1' # Looks like you failed 3 tests of 4. NOTE: Without these patches, the same test passes with me.
(In reply to Marcel de Rooy from comment #10) > Running tests (1) > * Proving /usr/share/koha/t/db_dependent/Authority/Merge.t KO! So it's strange. I don't know... I am helpless. Maybe you could have a look at the test -- is ModBiblio properly mocked in the test snippet added to Merge.t?
I now had this one: # Subtest: Test merge A1 to modified A1, test strict mode 1..12 not ok 1 - Both records are updated now # Failed test 'Both records are updated now' # at t/db_dependent/Authority/Merge.t line 174. # got: '1' # expected: '2' And that is an existing test. Without your patch it passes. Please have another look here.
And the following ones are new (iiuc): # Subtest: ModBiblio calls from merge 1..4 not ok 1 - real modification of bibliographic record, ModBiblio called # Failed test 'real modification of bibliographic record, ModBiblio called' # at t/db_dependent/Authority/Merge.t line 716. # didn't find a warning # expected to find warning: ModBiblio called not ok 2 - no real modification of bibliographic record, index_records called # Failed test 'no real modification of bibliographic record, index_records called' # at t/db_dependent/Authority/Merge.t line 730. # didn't find a warning # expected to find warning: index_records called ok 3 - no real modification of bibliographic record, index_records not called not ok 4 - Only one bibliographic record updated while merging two authorities # Failed test 'Only one bibliographic record updated while merging two authorities' # at t/db_dependent/Authority/Merge.t line 759. # got: '0' # expected: '1' # Looks like you failed 3 tests of 4. That could indeed be a bad mock or so ?
+ my $biblio_module = Test::MockModule->new('C4::AuthoritiesMarc'); This is confusing btw :)
(In reply to Marcel de Rooy from comment #14) > + my $biblio_module = Test::MockModule->new('C4::AuthoritiesMarc'); > > This is confusing btw :) Maybe a little... This snippet is taken from one of the previous tests (cf. commit 3954808ac5052a53d5fa40502a4f7bd991be80ab).
Created attachment 179979 [details] [review] Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Created attachment 179980 [details] [review] Bug 34739: Unit tests BTW, there is a tiny adjustment to one of the previous subtests to fit to the modified merge behaviour. Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
(In reply to Marcel de Rooy from comment #14) > + my $biblio_module = Test::MockModule->new('C4::AuthoritiesMarc'); > > This is confusing btw :) Loading authorities and calling it biblio, I mean specifically :)
I'm not sure how expensive calling C4::Context->preference('IncludeSeeFromInSearches') is, but probably it would be better fetch both sysprefs once, put them into a variable and inside the loop just access those vars: so instead of if ( !$update && $reindex_if_needed && ( C4::Context->preference('IncludeSeeFromInSearches') || C4::Context->preference('IncludeSeeAlsoFromInSearches') ) ) do my $syspref_include_see_from = C4::Context->preference('IncludeSeeFromInSearches') || C4::Context->preference('IncludeSeeAlsoFromInSearches') || 0; while ( my $biblio = $biblios->next ) { ... if ( !$update && $reindex_if_needed && $syspref_include_see ) Also maybe $update should be renamed to $update_biblio?
Created attachment 180083 [details] [review] Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
(In reply to Thomas Klausner from comment #19) > I'm not sure how expensive calling > C4::Context->preference('IncludeSeeFromInSearches') is, but probably it > would be better fetch both sysprefs once, put them into a variable and > inside the loop just access those vars: OK, corrected according to your suggestion. I wanted to make the checking conditional but at the end it should be far better to check in once and remember. > Also maybe $update should be renamed to $update_biblio? This comes from the old code -- I would prefer not to touch the existing code if not necessary.
Created attachment 180187 [details] [review] Bug 34739: Unit tests BTW, there is a tiny adjustment to one of the previous subtests to fit to the modified merge behaviour. Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> Signed-off-by: Thomas Klausner <domm@plix.at>
Created attachment 180188 [details] [review] Bug 34739: Linked biblios should not be merged (updated) when changes to an authority don't change the authorized heading Currently, the function C4::AuthoritiesMarc::merge calls ModBiblio for all bibliographic records linked to the modified authority record. It could seem a just behavior but we know that this could lead to performance problems. This is why AuthorityMergeLimit systempreference with merge_authorities.pl was introduced (earlier: dontmerge). Although this partially solves the performance problems, for moderately large and large catalogs (500+ K) and authority records linked with a large number of bibliographic records, performance problems still occur, even with overnight processing with merge_authorities.pl. This problem is particularly apparent when updating authority records from central databases, whose modifications are beyond our control. It turns out, however, that many such modifications to authority records involve fields other than the heading field (1XX in MARC 21), so these updates in fact do not modify bibliographic records. However, with the current merge function, every touch on an authority record forces a call to the ModBiblio function. The number of relatively expensive ModBiblio calls could be reduced by checking whether there has been an actual change in the field associated with the authority record. However, with IncludeSeeFromInSearches or IncludeSeeAlsoFromInSearches preferences enabled, the record should be refreshed in the search engine just in case (should the 4XX, 5XX fields have changed, which we don't know at this point). Test plan: ========== 1. Make sure to have CataloguingLog set to Log and IncludeSeeFromInSearches set to Include. 2. Have a bibliographic record with an authority record linked. Open the authority record in a separate tab, open it in the editor and save, without modifying it. Repeat this 2-3 times. In the bibliographic record tab open the Modification log and see a number of entries - for each “empty” modification of the linked authority record. 3. Apply the patch ; restart_all. 4. Repat p. 2. Note that there are no new entries in the log. 5. Edit the authority record again, this time putting in one of the proper 4XX $a subfield an uncommon word (but do not touch the 1XX field). Save the authority record. In the bibliographic record tab control that there are no new entries in the modification log. Make a search for the word put in 4XX field. You should get the bibliographic record(s) linked to the edited authority record, even though the bibliographic record itself has not been modified. 6. Edit the authority record again, this time changing the 1XX field. Save the authority record. In the bibliographic record tab control that there are is new entry in the modification log. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl> Signed-off-by: Thomas Klausner <domm@plix.at>
Work! Very nice, this will remove a lot of unnecessary updates on biblios.
Pushed for 25.05! Well done everyone, thank you!
We have some broken tests, please check ASAP (lots waiting to be pushed): Subtest: ModBiblio calls from merge 1..5 not ok 1 - real modification of bibliographic record, ModBiblio called not ok 2 - no real modification of bibliographic record, index_records called ok 3 - no real modification of bibliographic record, index_records not called not ok 4 - real modification of bibliographic record, ModBiblio called not ok 5 - only one bibliographic record updated while merging two authorities
Note: I suspect it comes from this patch set as it relates to merges, but please verify.
when I run prove -lv t/db_dependent/Biblio_holdsqueue.t or t/db_dependent/Biblio.t and then t/db_dependent/Authority/Merge.t I get the failures. If then do reset_all, Merge.t works again. But I haven't had time to take a closer look why this happens
If you run t/db_dependent/Biblio.t first, then Merge.t will fail. I guess there are modifications made by Biblio.t to the MSS (marc subfield structure), either in DB or cached. I have noticed this suspicious stuff: 698 $biblio_record->insert_fields_ordered( 699 MARC::Field->new( '609', '1', '1', a => 'Brown,', 'c' => 'Father', 9 => $authid ) ); 700 my $biblionumber = AddBiblio( $biblio_record, '', { skip_record_index => 1 } ); if you dump $biblio_record->as_formatted you see the 609 field. BUT, if you add the following line right after: my $test = Koha::Biblios->find($biblionumber)->metadata->record then dump $test->as_formatted => 609 is not there (??)
(In reply to Jonathan Druart from comment #29) > If you run t/db_dependent/Biblio.t first, then Merge.t will fail. > > I guess there are modifications made by Biblio.t to the MSS (marc subfield > structure), either in DB or cached. > > I have noticed this suspicious stuff: > > 698 $biblio_record->insert_fields_ordered( > 699 MARC::Field->new( '609', '1', '1', a => 'Brown,', 'c' => > 'Father', 9 => $authid ) ); > 700 my $biblionumber = AddBiblio( $biblio_record, '', { > skip_record_index => 1 } ); > > if you dump $biblio_record->as_formatted you see the 609 field. > > BUT, if you add the following line right after: > my $test = Koha::Biblios->find($biblionumber)->metadata->record > then dump $test->as_formatted > > => 609 is not there (??) Sounds like a framework issue.
The new test case calls AddBiblio. And AddBiblio returns ($biblionumber, $biblioitemnumber). But the test calls it like this: my $biblionumber = AddBiblio; AddBiblio returns a list, but the calling code only stores a single value, so it get's the last value, i.e. the biblioitemnumber.  Now, when running the test alone, the DB sequences for biblio and biblioitem are in sync and both return the same value so the test works (by accident) But some tests add biblios with multiple items (or biblios without items), thus the sequences are no longer in sync and when the test tries to load a biblio via a biblioitemid, it finds nothing, which causes the fail bad: my $biblionumber = AddBiblio( $biblio_record, '', { skip_record_index => 1 } ); ok-ish: my ($biblionumber) = AddBiblio( $biblio_record, '', { skip_record_index => 1 } );
See Bug 39700 for the fix
My fault :( Thanks, Thomas!
Fixed on separate bug, thanks!