Bug 35125 - AutoCreateAuthorities creates separate authorities when thesaurus differs, even with LinkerConsiderThesaurus set to Don't
Summary: AutoCreateAuthorities creates separate authorities when thesaurus differs, ev...
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low major
Assignee: Nick Clemens (kidclamp)
QA Contact: Martin Renvoize (ashimema)
URL:
Keywords:
Depends on: 30280
Blocks:
  Show dependency treegraph
 
Reported: 2023-10-21 00:20 UTC by Phil Ringnalda
Modified: 2024-11-15 15:56 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
24.11.00,24.05.05,23.11.11
Circulation function:


Attachments
Bug 35125: Don't use thesaurus value in cache when LinkerConsiderThesaurus is set to don't (4.75 KB, patch)
2024-06-24 20:04 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Test bibs (851 bytes, application/marc)
2024-06-25 02:37 UTC, Phil Ringnalda
Details
Bug 35125: same adjustments for update_cache (2.23 KB, patch)
2024-06-25 14:56 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 35125: Don't use thesaurus value in cache when LinkerConsiderThesaurus is set to don't (4.80 KB, patch)
2024-06-25 20:26 UTC, Phil Ringnalda
Details | Diff | Splinter Review
Bug 35125: same adjustments for update_cache (2.28 KB, patch)
2024-06-25 20:26 UTC, Phil Ringnalda
Details | Diff | Splinter Review
Bug 35125: Respect LinkerConsiderThesaurus in get_link (4.84 KB, patch)
2024-06-28 15:19 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 35125: Respect LinkerConsiderThesaurus in update_cache (2.36 KB, patch)
2024-06-28 15:19 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Phil Ringnalda 2023-10-21 00:20:29 UTC
Setup:

* Set AutoCreateAuthorities to generate, AutoLinkBiblios to Do, CatalogModuleRelink to Do (and  EnableAdvancedCatalogingEditor to enable if you want to make your life easier)
* Ensure that LinkerConsiderThesaurus is at the default Don't
* Make sure you don't have a genre authority record for Boring stories

Add the following headings to a bib record and save it

655 _ _ ‡aBoring stories
655 _ 0 ‡aBoring stories
655 _ 4 ‡aBoring stories
655 _ 7 ‡aBoring stories‡2gsafd
655 _ 7 ‡aBoring stories‡2lcgft

Note that you now have each heading linked to a different authid, and you have five genre authority records for Boring stories, all identical.

If instead you already had a Boring stories authority record (no matter the thesaurus) before saving the record, they would have all been linked to it.
Comment 1 Phil Ringnalda 2023-10-21 00:26:24 UTC
Seems like it must be from the linker caching with $thesaurus in the key, and only Heading knowing that LinkerConsiderThesaurus is Don't, but it's not obvious why.
Comment 2 Phil Ringnalda 2024-06-22 23:15:57 UTC
Still does, still annoying having two name authorities created for every autobiography where the author is also a subject.
Comment 3 Esther Melander 2024-06-24 19:19:12 UTC
The example authorities all differ in some way, so is the linker checker taking into consideration the indicators and subfields? One could argue an authority with a second indicator 4 is different from one with a second indicator 7.
Comment 4 Phil Ringnalda 2024-06-24 20:00:26 UTC
They are different, and with LinkerConsiderThesaurus set to Do and AutoCreateAuthorities set to don't generate they will be treated as being different.

With LinkerConsiderThesaurus set to Do and AutoCreateAuthorities set to generate, they will all be treated as being different but AutoCreateAuthorities will create identical authorities for them, which is why I filed bug 35125 to make it possible to use autocreate in a considerthesaurus world, and then that was why Nick filed bug 33557 to have a preference to shut off considerthesaurus, because otherwise without a fix for bug 35125 autocreate would be a nightmare.

But with LinkerConsiderThesaurus: Don't and AutoCreateAuthorities generate, AutoCreateAuthorities will create five identical authority records for those five things, even though if you already had one authority record the linker would have linked all five to just that one. That's this bug.
Comment 5 Nick Clemens (kidclamp) 2024-06-24 20:04:45 UTC
Created attachment 168040 [details] [review]
Bug 35125: Don't use thesaurus value in cache when LinkerConsiderThesaurus is set to don't

This patch adds a new key 'notconsidered' to the authority cache when linking to avoid doubling when creating new authorities and wishing to ignore thesaurus

To test:
0 - Set LinkerConsiderThesaurus to "don't"
1 - Set AutCreateAuthorities to generate
2 - Set AutLinkBiblios to Do
3 - Set CataloguingModuleRelink to Do
4 - Save a new record in Koha with the same radnomized heading repeated but from 3 different authority sources
    ensure source is defined by indicators only and by field $2 once
    ensure the heading is random so that no matches will be found
5 - Confirm each authority is linked to a new unique authority
6 - Apply patch, restart all
7 - Repeat 4 with a new subject heading
8 - Confirm all headings are linked to the same authority
Comment 6 Nick Clemens (kidclamp) 2024-06-24 20:05:46 UTC
 I added unit tests and they pass with this, but I didn't test in the interface - if you want to flesh out the test plan more please do and I'll look again if it doesn't work
Comment 7 Phil Ringnalda 2024-06-24 20:22:23 UTC
100 and 600 is the only other flavor I've seen, I'll test that when I get time to test. Thanks for the patch!
Comment 8 Phil Ringnalda 2024-06-24 22:50:01 UTC
Tested in a sandbox and the patch made no difference. No reason it shouldn't work in a sandbox, so I'm not hopeful.
Comment 9 Phil Ringnalda 2024-06-25 02:03:59 UTC
Alas, the patch doesn't work, AutoCreateAuthorities still creates separate authorities for each repeated subject.
Comment 10 Phil Ringnalda 2024-06-25 02:37:13 UTC
Created attachment 168047 [details]
Test bibs

Conveniently, batch import shows the same bug. Be more convenient if you could import records in a batch individually, though. I was going to make a batch of twenty different records, but having twenty separate batches would just be a pain to keep straight.

Interesting behavior (though I'm not sure it's actually A Clue): if you import the batch, then import again not looking for matches so you import duplicates, the duplicates will link exactly like the first records did: the first Subject One linked to 1710 in both records, the second to 1711 in both records, etc. I guess that just means that the cache still knows that they are different, and how they are different.
Comment 11 Phil Ringnalda 2024-06-25 03:07:39 UTC
Both get_link and update_cache need to be setting 'notconsidered', but that's not enough to make it work.
Comment 12 Nick Clemens (kidclamp) 2024-06-25 14:56:28 UTC
Created attachment 168078 [details] [review]
Bug 35125: same adjustments for update_cache
Comment 13 Phil Ringnalda 2024-06-25 15:01:00 UTC
Doesn't update_cache need to be setting match_count, so we don't still have a match_count of zero after AutoCreateAuthorities created one and updated the cache?
Comment 14 Phil Ringnalda 2024-06-25 15:01:23 UTC
As in, "how did that ever work?"
Comment 15 Nick Clemens (kidclamp) 2024-06-25 15:11:26 UTC
(In reply to Phil Ringnalda from comment #13)
> Doesn't update_cache need to be setting match_count, so we don't still have
> a match_count of zero after AutoCreateAuthorities created one and updated
> the cache?

Yes, but I think match_count only comes in to play if there is no authid.

In this case, we set the authid - which means that whatever linker we are using found a positive match. 

The count is for when we didn't find a match and need to know if we should create or not.
Comment 16 Phil Ringnalda 2024-06-25 16:28:30 UTC
Comment on attachment 168047 [details]
Test bibs

Oops, uploaded the pre-linked file, not the unlinked one, that leads to baffling results.
Comment 17 Phil Ringnalda 2024-06-25 20:23:41 UTC
And apparently I'm confused about how batch import works, and when I see authority records created from batch imports in my production system, it's because of people or cron jobs doing other things to them, so comment 11 was just from importing a wrongly pre-linked record and not realizing it was pre-linked and not relinked or freshly linked.
Comment 18 Phil Ringnalda 2024-06-25 20:26:28 UTC
Created attachment 168096 [details] [review]
Bug 35125: Don't use thesaurus value in cache when LinkerConsiderThesaurus is set to don't

This patch adds a new key 'notconsidered' to the authority cache when linking to avoid doubling when creating new authorities and wishing to ignore thesaurus

To test:
0 - Set LinkerConsiderThesaurus to "don't"
1 - Set AutCreateAuthorities to generate
2 - Set AutLinkBiblios to Do
3 - Set CataloguingModuleRelink to Do
4 - Save a new record in Koha with the same radnomized heading repeated but from 3 different authority sources
    ensure source is defined by indicators only and by field $2 once
    ensure the heading is random so that no matches will be found
5 - Confirm each authority is linked to a new unique authority
6 - Apply patch, restart all
7 - Repeat 4 with a new subject heading
8 - Confirm all headings are linked to the same authority

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Comment 19 Phil Ringnalda 2024-06-25 20:26:30 UTC
Created attachment 168097 [details] [review]
Bug 35125: same adjustments for update_cache

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Comment 20 Martin Renvoize (ashimema) 2024-06-28 15:19:06 UTC
Created attachment 168261 [details] [review]
Bug 35125: Respect LinkerConsiderThesaurus in get_link

This patch adds a new key 'notconsidered' to the authority cache when
linking to avoid doubling when creating new authorities and wishing
to ignore thesaurus

To test:
0 - Set LinkerConsiderThesaurus to "don't"
1 - Set AutCreateAuthorities to generate
2 - Set AutLinkBiblios to Do
3 - Set CataloguingModuleRelink to Do
4 - Save a new record in Koha with the same randomized heading repeated
    but from 3 different authority sources
    ensure source is defined by indicators only and by field $2 once
    ensure the heading is random so that no matches will be found
5 - Confirm each authority is linked to a new unique authority
6 - Apply patch, restart all
7 - Repeat 4 with a new subject heading
8 - Confirm all headings are linked to the same authority

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 21 Martin Renvoize (ashimema) 2024-06-28 15:19:09 UTC
Created attachment 168262 [details] [review]
Bug 35125: Respect LinkerConsiderThesaurus in update_cache

Signed-off-by: Phil Ringnalda <phil@chetcolibrary.org>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 22 Martin Renvoize (ashimema) 2024-06-28 15:21:56 UTC
All works as expected now, thanks.

Passing QA
Comment 23 Katrin Fischer 2024-07-01 16:58:15 UTC
Pushed for 24.11!

Well done everyone, thank you!
Comment 24 Lucas Gass (lukeg) 2024-10-25 21:37:14 UTC
Backported to 24.05.x for upcoming 24.05.05
Comment 25 Fridolin Somers 2024-11-15 15:56:50 UTC
Pushed to 23.11.x for 23.11.11