Bug 39503 - Linker should always respect thesaurus with LinkerConsiderThesaurus on
Summary: Linker should always respect thesaurus with LinkerConsiderThesaurus on
Status: Pushed to main
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low normal
Assignee: Janusz Kaczmarek
QA Contact: Martin Renvoize (ashimema)
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-03-31 14:38 UTC by Janusz Kaczmarek
Modified: 2025-05-23 11:33 UTC (History)
4 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
25.05.00
Circulation function:


Attachments
Bug 39503: Bug 39503 - Linker should always respect thesaurus with LinkerConsiderThesaurus on (3.63 KB, patch)
2025-03-31 19:33 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 39503: Unit tests (5.82 KB, patch)
2025-04-01 11:24 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 39503: Bug 39503 - Linker should always respect thesaurus with LinkerConsiderThesaurus on (3.68 KB, patch)
2025-04-01 14:25 UTC, Roman Dolny
Details | Diff | Splinter Review
Bug 39503: Unit tests (5.87 KB, patch)
2025-04-01 14:25 UTC, Roman Dolny
Details | Diff | Splinter Review
Bug 39503: [alternate] Collapse thesaurus if value in 008/_11 and 040 (1.26 KB, patch)
2025-04-11 12:36 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 39503: Collapse thesaurus if value in 008/_11 and 040 (3.22 KB, patch)
2025-04-14 11:37 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 39503: Unit tests (6.54 KB, patch)
2025-04-14 11:37 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 39503: Collapse thesaurus if value in 008_/11 = 'z' and 040 $f (3.32 KB, patch)
2025-04-14 20:26 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 39503: Unit tests (9.99 KB, patch)
2025-04-14 20:26 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Bug 39503: Collapse thesaurus if value in 008_/11 = 'z' and 040 $f (3.37 KB, patch)
2025-04-19 09:06 UTC, Roman Dolny
Details | Diff | Splinter Review
Bug 39503: Unit tests (10.06 KB, patch)
2025-05-09 14:25 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 39503: Collapse thesaurus if value in 008_/11 = 'z' and 040 $f (3.44 KB, patch)
2025-05-09 14:25 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Janusz Kaczmarek 2025-03-31 14:38:31 UTC
With bug 30280 and subsequent patches we attempt to respect the thesaurus indication when linking with LinkerConsiderThesaurus on. However, there are still cases when we link wrong. 

Example:

biblio:

650 _1 $a Philosophy
[thesaurus: Library of Congress Children's and Young Adults' Subject Headings]

will be linked to an authority

008/11 = a  [Library of Congress Subject Headings]
150 __ $a Philosophy

if 

008/11 = b  [Library of Congress Children's and Young Adults' Subject Headings]
150 __ $a Philosophy

is missing in the system. This is obviously wrong--we should not link to a authority record belonging to a different thesaurus.
Comment 1 Janusz Kaczmarek 2025-03-31 14:51:44 UTC
I put wrong example. Actually, the problem arises only with thesauri defined in with 008/11 = 'z' and 040 $f.

Example:

biblio:

650 _7 $a Filozofia $2 dbn
[thesaurus: dbn]

will be linked to an authority

008/11 = z  
040 $f kaba
150 __ $a Filozofia

if authority record:

008/11 = z
040 $f dbn
150 __ $a Filozofia

is missing in the system. This is obviously wrong--we should not link to a authority record belonging to a different thesaurus.
Comment 2 Janusz Kaczmarek 2025-03-31 19:18:55 UTC
This happens when thesaurus in bibliographic field 6XX is defined by subfield $2. In such a case the authority record should have 008/11 = 'z' and, according to https://www.loc.gov/marc/authority/ad008.html (cf. comment for 008/11 = 'z': "A MARC code for the conventions used to formulate the heading *may* be contained in subfield $f"), may or may have thesaurus declared in 040 $f. 

This is why we make a second search attempt in C4::Heading::_search, but doing so we have to make sure that the resulting authority record has not a different thesaurus declared in 040 $f. (Note that 008/11 and 040 $f are indexed with  the same search field: subject-heading-thesaurus. So we search for the second time in the same search field but with 'notdefined' == 008/11='z' instead of the original thesaurus taken from subfield $2 of the bibliographic field 6XX.) This means we have to explicitly check the content of 040 $f of the resulting authority record.
Comment 3 Janusz Kaczmarek 2025-03-31 19:33:28 UTC
Created attachment 180082 [details] [review]
Bug 39503:  Bug 39503 - Linker should always respect thesaurus with LinkerConsiderThesaurus on

With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.

This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.

This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with  the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.

Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
   040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
   in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
   the term 'Early music' in $a, and a different thesaurus code in $2
   (e.g. 'dbn'). Click the 'Link authorities automatically' button.
   Note that 650 'Early music' (dbn) has been linked with and authority
   record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all.
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
   an authority record (650 - No matching authority found.)
Comment 4 Janusz Kaczmarek 2025-04-01 11:24:28 UTC
Created attachment 180131 [details] [review]
Bug 39503: Unit tests

NB, some previous test had to be adjusted to make it posible to call
search_auth_compat in _search and properly interpret its results.
Comment 5 Roman Dolny 2025-04-01 14:25:23 UTC
Created attachment 180200 [details] [review]
Bug 39503: Bug 39503 - Linker should always respect thesaurus with LinkerConsiderThesaurus on

With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.

This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.

This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with  the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.

Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
   040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
   in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
   the term 'Early music' in $a, and a different thesaurus code in $2
   (e.g. 'dbn'). Click the 'Link authorities automatically' button.
   Note that 650 'Early music' (dbn) has been linked with and authority
   record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all.
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
   an authority record (650 - No matching authority found.)

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Comment 6 Roman Dolny 2025-04-01 14:25:26 UTC
Created attachment 180201 [details] [review]
Bug 39503: Unit tests

NB, some previous test had to be adjusted to make it posible to call
search_auth_compat in _search and properly interpret its results.

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Comment 7 Nick Clemens (kidclamp) 2025-04-11 12:36:58 UTC
Created attachment 180844 [details] [review]
Bug 39503: [alternate] Collapse thesaurus if value in 008/_11 and 040
Comment 8 Janusz Kaczmarek 2025-04-14 11:37:09 UTC
Created attachment 180902 [details] [review]
Bug 39503: Collapse thesaurus if value in 008/_11 and 040

With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.

This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.

This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with  the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.

Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
   040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
   in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
   the term 'Early music' in $a, and a different thesaurus code in $2
   (e.g. 'dbn'). Click the 'Link authorities automatically' button.
   Note that 650 'Early music' (dbn) has been linked with and authority
   record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
   koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
   an authority record (650 - No matching authority found.)

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 9 Janusz Kaczmarek 2025-04-14 11:37:12 UTC
Created attachment 180903 [details] [review]
Bug 39503: Unit tests

NB, some previous test had to be adjusted to make it posible to call
search_auth_compat in _search and properly interpret its results.

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Comment 10 Janusz Kaczmarek 2025-04-14 20:26:37 UTC
Created attachment 180940 [details] [review]
Bug 39503: Collapse thesaurus if value in 008_/11 = 'z' and 040 $f

With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.

This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.

This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with  the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.

Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
   040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
   in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
   the term 'Early music' in $a, and a different thesaurus code in $2
   (e.g. 'dbn'). Click the 'Link authorities automatically' button.
   Note that 650 'Early music' (dbn) has been linked with and authority
   record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
   koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
   an authority record (650 - No matching authority found.)

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Comment 11 Janusz Kaczmarek 2025-04-14 20:26:41 UTC
Created attachment 180941 [details] [review]
Bug 39503: Unit tests

NB, some previous test had to be adjusted to make it posible to call
search_auth_compat in _search and properly interpret its results.

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Comment 12 Roman Dolny 2025-04-19 09:06:35 UTC
Created attachment 181205 [details] [review]
Bug 39503: Collapse thesaurus if value in 008_/11 = 'z' and 040 $f

With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.

This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.

This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with  the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.

Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
   040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
   in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
   the term 'Early music' in $a, and a different thesaurus code in $2
   (e.g. 'dbn'). Click the 'Link authorities automatically' button.
   Note that 650 'Early music' (dbn) has been linked with and authority
   record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
   koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
   an authority record (650 - No matching authority found.)

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Comment 13 Martin Renvoize (ashimema) 2025-05-09 14:25:38 UTC
Created attachment 182182 [details] [review]
Bug 39503: Unit tests

NB, some previous test had to be adjusted to make it posible to call
search_auth_compat in _search and properly interpret its results.

Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 14 Martin Renvoize (ashimema) 2025-05-09 14:25:41 UTC
Created attachment 182183 [details] [review]
Bug 39503: Collapse thesaurus if value in 008_/11 = 'z' and 040 $f

With bug 30280 and subsequent patches we attempt to respect the
thesaurus indication when linking with LinkerConsiderThesaurus on.
However, there are still cases when we link wrong.

This happens when thesaurus in bibliographic field 6XX is defined by
subfield $2. In such a case the authority record should have 008/11 = 'z'
and, according to https://www.loc.gov/marc/authority/ad008.html
(cf. comment for 008/11 = 'z': "A MARC code for the conventions used to
formulate the heading *may* be contained in subfield $f"), may or may have
thesaurus declared in 040 $f.

This is why we make a second search attempt in C4::Heading::_search,
but doing so we have to make sure that the resulting authority
record has not a different thesaurus declared in 040 $f. (Note
that 008/11 and 040 $f are indexed with  the same search
field: subject-heading-thesaurus. So we search for the
second time in the same search field but with
'notdefined' == 008/11='z' instead of the original
thesaurus taken from subfield $2 of the bibliographic
field 6XX.) This means we have to explicitly check the
content of 040 $f of the resulting authority record.

Test plan:
==========
1. Turn on the LinkerConsiderThesaurus systempreference.
2. Create an authority record (e.g. TOPIC_TERM) with 008/11 = 'z' and
   040 $f containing a thesaurus indication (e.g. 'kaba'), and a term
   in field 150 $a (e.g. Early music).
3. Edit a bibliographic record: in field 650 put 2nd indicator '7',
   the term 'Early music' in $a, and a different thesaurus code in $2
   (e.g. 'dbn'). Click the 'Link authorities automatically' button.
   Note that 650 'Early music' (dbn) has been linked with and authority
   record 'Early music' (kaba). This is wrong.
4. Apply the patch ; restart_all ; reindex ES for authorities with:
   koha-elasticsearch --rebuild -d -a kohadev
5. Repeat p. 3. Now the 650 'Early music' (dbn) has not been linked to
   an authority record (650 - No matching authority found.)

Signed-off-by: Janusz Kaczmarek <januszop@gmail.com>
Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 15 Katrin Fischer 2025-05-13 16:05:22 UTC
The bug description mentions LinkerConsiderThesaurus, but I don't spot any code checking for the preference in the patch set?
Comment 16 Katrin Fischer 2025-05-13 16:47:48 UTC
Pushed for 25.05!

Well done everyone, thank you!
Comment 17 Janusz Kaczmarek 2025-05-23 11:33:24 UTC
(In reply to Katrin Fischer from comment #15)
> The bug description mentions LinkerConsiderThesaurus, but I don't spot any
> code checking for the preference in the patch set?

This is because my original version of the patch (https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=180082&action=diff) referred to C4/Heading.pm (and the snippet was executed only if LinkerConsiderThesaurus was set).

Now, the bug could be named after the patch title, like "Collapse elasticsearch subject-heading-thesaurus search field if value in 008_/11 = 'z' and 040 $f" or somehow similar.