Bug 32916 - [Bug 30280 follow-up] Problems in linking authorities to biblio fields (MARC 21)
Summary: [Bug 30280 follow-up] Problems in linking authorities to biblio fields (MARC 21)
Status: RESOLVED DUPLICATE of bug 33277
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: 22.11
Hardware: All All
: P2 major (vote)
Assignee: Janusz Kaczmarek
QA Contact: Testopia
URL:
Keywords:
Depends on: 30280
Blocks:
  Show dependency treegraph
 
Reported: 2023-02-08 19:05 UTC by Janusz Kaczmarek
Modified: 2023-05-04 10:57 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
[PATCH] Bug 32916: [Bug 30280 follow-up] Problems in linking authorities to biblio fields (MARC 21) (15.64 KB, patch)
2023-02-10 22:30 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
Sample data for testing (1.28 KB, application/x-compressed)
2023-02-10 22:31 UTC, Janusz Kaczmarek
Details
[PATCH] Bug 32916: [Bug 30280 follow-up] Problems in linking authorities to biblio fields (MARC 21) (15.37 KB, patch)
2023-02-13 19:27 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review
[PATCH] Bug 32916: [Bug 30280 follow-up] Problems in linking authorities to biblio fields (MARC 21) - Zebra mangling (15.68 KB, patch)
2023-02-13 19:31 UTC, Janusz Kaczmarek
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Janusz Kaczmarek 2023-02-08 19:05:21 UTC
I encounter two problems connected to recent changes in the Linker module:

1.	Problem with thesaurus in biblio fields different than 6XX (Zebra and ES)

a)	The search for a needed authority record (in C4::Heading::_search) is made now not only with the search form derived from controlled field content, but also with the second operand – the thesaurus.  The second operand is added when $self->{thesaurus} is defined.  And at the current for MARC 21 it is always defined (C4::Heading::MARC21::parse_heading returns the right thesaurus symbol for 6XX fields and ‘lcsh’ for the rest, e.g. 1XX, 7XX, 8XX).

b)	So, if $self->{thesaurus} ide defined (= always), the thesaurus operand in authority search is determined by examining the second indicator of the field in the bibliographic record. (I leave apart the discussion of why it is taken from the internal structure of the MARC::Field object while $self->{field}->indicator(‘2’) would be the correct way). But the second indicator makes sense – in terms of thesaurus – only for fields 6XX.  For other controlled fields it is either undefined or has completely different meaning (e.g. in 700, 710, 711, 730, 830 etc.).  The 

c)	So, in C4::Heading::_search we should consider the thesaurus operand only for 6XX fields. This could be done either by checking $self->{field}->tag in C4::Heading::_search or by not defining $self->{thesaurus} in C4::Heading::MARC21::parse_heading for fields != 6XX.

2.	Problem with thesauri in Zebra searches

a)	While preparing Zebra indexes for authorities (authority-zebra-indexdefs.xsl), the 008/11 byte or authority record is transformed from one character code to the symbol (a => lcsh, b => lcac, etc.).

b)	But in C4::Heading::_search the thesaurus operand is formulated as one character code conforming to the auth 008/11 convention. As a consequence, no results are returned (unless by accident). 

c)	A possible solution could be removing the transformation (authority-zebra-indexdefs.xsl, koha-indexdefs-to-zebra.xsl) while generating Zebra index or refactoring the thesaurus fragment in C4::Heading::_search to reflect this duality in recording information in indexes (Zebra vs. ES).

Could anybody confirm that the problems occur in other installations?

Cheers! - Janusz
Comment 1 Janusz Kaczmarek 2023-02-10 22:30:46 UTC
Created attachment 146523 [details] [review]
[PATCH] Bug 32916: [Bug 30280 follow-up] Problems in linking  authorities to biblio fields (MARC 21)

After applying the bug patch 30280:

1. Koha does not link with Zebra.

2. Koha does not properly link headings other than 6XX with ES.

3. Koha does not link subject headings when 6XX indicator 2 = ‘4’ or in
the case when in the auth record 008/11 = ‘z’, but 040 $f is not
defined, which is legal in terms of MARC 21 documentation
(https://www.loc.gov/marc/authority/ad008.html): “A MARC code for the
conventions used to formulate the heading may be contained in
subfield $f (Subject heading/thesaurus conventions) in field 040
(Cataloging Source).” -- ‘may’, not ‘should’.

A possible solution to this is to make ES emulate the Zebra indexing
of auth 008/11 and a correction to C4::Heading::_search.  Some minor
corrections in other places had to be done.

This solution requires a new ES field type and so also a modification
of the database – an expansion of the ENUM type for search_field.type
(ALTER TABLE search_field MODIFY COLUMN `type`
ENUM('','string','date','number','boolean','sum','isbn','stdno','year','callnumber','thesaurus')
NOT NULL COMMENT 'what type of data this holds, relevant when storing
it in the search engine';)

The new strict behaviour should be controlled (on/off) by a new
preference (e.g. LinkerStrictAuthInfo) -- since not every library is
using well formatted data.  The semantics of this preference could be
expanded in the future to take into account also 008/14-16 of an auth
record for instance.

Test plan:
==========

1. Have a clean master (or 22.11.0[0-2]) installation

2. Load provided data sample (bulkmarcimport.pl -d -a/b -file ...)

3. Reindex with Zebra and ES

4. Perform link_bibs_to_authorities.pl -v -t -l with Zebra, you
should get:

Linked headings (from most frequent to least):
-------------------------------------------------------

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       3 occurrences
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject lcsh:   1 occurrences
Subject with thesaurus not specified:   1 occurrences

5. Perform link_bibs_to_authorities.pl -v -t -l with ES, you
should get:

Linked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       2 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Subject lcsh:   1 occurrences

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
Feminism:       1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject with thesaurus not specified:   1 occurrences

6. Apply the patch (pay attention to the location of the
authority-zebra-indexdefs.xsl file in you test environment).
Add a system preference LinkerStrictAuthInfo = 1 and perform
database modification (ALTER TABLE search_field MODIFY
COLUMN `type`
ENUM('','string','date','number','boolean','sum','isbn','stdno','year','callnumber','thesaurus')
NOT NULL COMMENT 'what type of data this holds, relevant
when storing it in the search engine';)

7. Full reindex with Zebra (koha-rebuild-zebra --full --force
-a -b) and ES (koha-elasticsearch --rebuild -r -d -a -b)

8. Perform link_bibs_to_authorities.pl -v -t -l with
Zebra and ES, you should get in both cases:

Linked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       2 occurrences
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject lcsh:   1 occurrences
Subject with thesaurus not specified:   1 occurrences

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       1 occurrences

9. Control the results in Koha -- all the heading
fields should be properly linked to the appropriate
auth records.
Comment 2 Janusz Kaczmarek 2023-02-10 22:31:46 UTC
Created attachment 146524 [details]
Sample data for testing
Comment 3 Janusz Kaczmarek 2023-02-10 22:51:43 UTC
The provided patch is just a proposal -- one of the possible scenarios to solve the observed malfunctioning.  I admit, it is quite complicated because of the introduction of a new ES type (which was necessary to be able to use the synonym token filter for 008/11).

Other possibility would be -- instead of digging into ES configuration to correct the Zebra index definition more aggressively (by introducing Subject-heading-thesaurus-conventions, not transforming 008/11 etc.), but also in this case a corrections to C4::Heading::_search and C4::Heading::MARC21::parse_heading along with _get_subject_thesaurus would be necessary. 

Yet another (probably less elegant) possibility would be to reflect in C4::Heading::_search the current difference in indexing 008/11 in Zebra and ES by creating a condition on C4::Context->preference('SearchEngine') and putting either 'a' or lcsh etc. as element of the array @value.

I would be curious to hear the opinions of those involved in the creation of the bug fix 30280.
Comment 4 Janusz Kaczmarek 2023-02-13 19:27:09 UTC
Created attachment 146598 [details] [review]
[PATCH] Bug 32916: [Bug 30280 follow-up] Problems in linking  authorities to biblio fields (MARC 21)

After applying the bug patch 30280:

1. Koha does not link with Zebra.

2. Koha does not properly link headings other than 6XX with ES.

3. Koha does not link subject headings when 6XX indicator 2 = "4" or in
the case when in the auth record 008/11 = "z", but 040 $f is not
defined, which is legal in terms of MARC 21 documentation
(https://www.loc.gov/marc/authority/ad008.html): "A MARC code for the
conventions used to formulate the heading may be contained in
subfield $f (Subject heading/thesaurus conventions) in field 040
(Cataloging Source)." -- "may", not "should".

A possible solution to this is to make ES emulate the Zebra indexing
of auth 008/11 and a correction to C4::Heading::_search.  Some minor
corrections in other places had to be done.

This solution requires a new ES field type and so also a modification
of the database – an expansion of the ENUM type for search_field.type
(ALTER TABLE search_field MODIFY COLUMN `type`
ENUM('','string','date','number','boolean','sum','isbn','stdno','year','callnumber','thesaurus')
NOT NULL COMMENT 'what type of data this holds, relevant when storing
it in the search engine';)

The new strict behaviour should be controlled (on/off) by a new
preference (e.g. LinkerStrictAuthInfo) -- since not every library is
using well formatted data.  The semantics of this preference could be
expanded in the future to take into account also 008/14-16 of an auth
record for instance.

Test plan:
==========

1. Have a clean master (or 22.11.0[0-2]) installation

2. Load provided data sample (bulkmarcimport.pl -d -a/b -file ...)

3. Reindex with Zebra and ES

4. Perform link_bibs_to_authorities.pl -v -t -l with Zebra, you
should get:

Linked headings (from most frequent to least):
-------------------------------------------------------

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       3 occurrences
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject lcsh:   1 occurrences
Subject with thesaurus not specified:   1 occurrences

5. Perform link_bibs_to_authorities.pl -v -t -l with ES, you
should get:

Linked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       2 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Subject lcsh:   1 occurrences

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
Feminism:       1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject with thesaurus not specified:   1 occurrences

6. Apply the patch (pay attention to the location of the
authority-zebra-indexdefs.xsl file in you test environment).
Add a system preference LinkerStrictAuthInfo = 1 and perform
database modification (ALTER TABLE search_field MODIFY
COLUMN `type`
ENUM('','string','date','number','boolean','sum','isbn','stdno','year','callnumber','thesaurus')
NOT NULL COMMENT 'what type of data this holds, relevant
when storing it in the search engine';)

7. Full reindex with Zebra (koha-rebuild-zebra --full --force
-a -b) and ES (koha-elasticsearch --rebuild -r -d -a -b)

8. Perform link_bibs_to_authorities.pl -v -t -l with
Zebra and ES, you should get in both cases:

Linked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       2 occurrences
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject lcsh:   1 occurrences
Subject with thesaurus not specified:   1 occurrences

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       1 occurrences

9. Perform link_bibs_to_authorities.pl (without -t) and
control the results in Koha -- all the heading fields
should be properly linked to the appropriate auth records.
Comment 5 Janusz Kaczmarek 2023-02-13 19:31:59 UTC
Created attachment 146599 [details] [review]
[PATCH] Bug 32916: [Bug 30280 follow-up] Problems in linking  authorities to biblio fields (MARC 21) - Zebra mangling

Here is an alternative version of the patch -- less ES changes, no DB corrections, more changes to Zebra and old Zebra-related code.  An additional system preference LinkerStrictAuthInfo still required in order to be able to switch off the more strict behaviour. 


After applying the bug patch 30280:

1. Koha does not link with Zebra.

2. Koha does not properly link headings other than 6XX with ES.

3. Koha does not link subject headings when 6XX indicator 2 = '4' or in
the case when in the auth record 008/11 = 'z', but 040 $f is not
defined, which is legal in terms of MARC 21 documentation
(https://www.loc.gov/marc/authority/ad008.html): "A MARC code for the
conventions used to formulate the heading may be contained in
subfield $f (Subject heading/thesaurus conventions) in field 040
(Cataloging Source)." -- 'may', not 'should'.

The new strict behaviour should be controlled (on/off) by a new
preference (e.g. LinkerStrictAuthInfo) -- since not every library is
using well formatted data.  The semantics of this preference could be
expanded in the future to take into account also 008/14-16 of an auth
record for instance.

Test plan:
==========

1. Have a clean master (or 22.11.0[0-2]) installation

2. Load provided data sample (bulkmarcimport.pl -d -a/b -file ...)

3. Reindex with Zebra and ES

4. Perform link_bibs_to_authorities.pl -v -t -l with Zebra, you
should get:

Linked headings (from most frequent to least):
-------------------------------------------------------

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       3 occurrences
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject lcsh:   1 occurrences
Subject with thesaurus not specified:   1 occurrences

5. Perform link_bibs_to_authorities.pl -v -t -l with ES, you
should get:

Linked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       2 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Subject lcsh:   1 occurrences

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
Feminism:       1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject with thesaurus not specified:   1 occurrences

6. Apply the patch (pay attention to the location of the
authority-zebra-indexdefs.xsl file in you test environment).
Add a system preference LinkerStrictAuthInfo = 1.

7. Full reindex with Zebra (koha-rebuild-zebra --full --force
-a -b) and ES (koha-elasticsearch --rebuild -r -d -a -b)

8. Perform link_bibs_to_authorities.pl -v -t -l with
Zebra and ES, you should get in both cases:

Linked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       2 occurrences
Author 01:      1 occurrences
Author 01. Second work title:   1 occurrences
feminism:       1 occurrences
Person DBN:     1 occurrences
Series entry:   1 occurrences
Subject DBN without non-mandatory 040f: 1 occurrences
Subject lcsh:   1 occurrences
Subject with thesaurus not specified:   1 occurrences

Unlinked headings (from most frequent to least):
-------------------------------------------------------
Feminism:       1 occurrences

9. Perform link_bibs_to_authorities.pl (without -t) and
control the results in Koha -- all the heading fields
should be properly linked to the appropriate auth records.
Comment 6 Frank Hansen 2023-02-20 15:06:52 UTC
(In reply to Janusz Kaczmarek from comment #3)
> 
> I would be curious to hear the opinions of those involved in the creation of
> the bug fix 30280.

The main purpose of fix 30280 is to make Koha to distinguish authority records for subject headings (6XX fields) based on thesaurus. The fix is ​​not made for non-6XX fields.

However, as you mention, Koha hardcodes non-6XX fields to 'lcsh' (which is equivalent to subject-heading-thesaurus = 'a'). This means that all authority records must coded as lcsh. So yes, we need to do something about it. According to https://www.loc.gov/marc/bibliographic/bdx00.html indicator 2 is undefined for 1XX, 7XX and 8XX fields. I saw you changed it to undef which is probably fine, unless it has other side effects.

So, we only care for thesaurus value for the 6XX fields and skip filtering on based on thesaurus for non-6XX fields. I have chosen to make a slightly different solution for this. In C4::Heading_search I check if indicator 2 has a value and is not blank. If it has a value, the filtering is performed on the thesaurus. But if it is empty, i.e. has a space, no filtering is performed. I made a fix for that, which I also mentioned in comment 58. I provide my fix as a comment so you can see how I made it. But it still doesn't solve the problem with Zebra and how it filters terms on the thesaurus.

My experience was that the thesaurus filtering in Zebra did not work after all, even though it was said to work. The same term from different thesauris were considered duplicates. So I wonder if something is broken here?

I really have no opinion on whether our thesaurus index should be called system-heading-thesaurus or system-heading-thesaurus-conventions. As long as it doesn't create confusion into thinking that the system-heading-thesaurus only contains 008/11, but also 040f.
Comment 7 Frank Hansen 2023-02-20 15:09:11 UTC
Here are the patch I made for C4::Heading_search:

diff --git a/C4/Heading.pm b/C4/Heading.pm
index 7541a56205..9b807b5337 100644
--- a/C4/Heading.pm
+++ b/C4/Heading.pm
@@ -224,8 +224,16 @@ sub _search {
             $subject_heading_thesaurus = 'k';
         } elsif ($ind2 eq '6') {
             $subject_heading_thesaurus = 'v';
-        } else {
+        } elsif ($ind2 eq '7') {
             $subject_heading_thesaurus = 'z';
+            push @marclist, 'thesaurus-conventions';
+            push @and_or, 'and';
+            push @excluding, '';
+            push @operator, 'is';
+            push @value, $self->{'thesaurus'};
+        } else {
+            # use lcsh for anything else
+            $subject_heading_thesaurus = 'a';
         }
         push @marclist, 'thesaurus';
         push @and_or, 'and';
@@ -234,14 +242,6 @@ sub _search {
         push @value, $subject_heading_thesaurus;
     }
 
-    if ($ind2 eq '7') {
-        push @marclist, 'thesaurus-conventions';
-        push @and_or, 'and';
-        push @excluding, '';
-        push @operator, 'is';
-        push @value, $self->{'thesaurus'};
-    }
-
     require Koha::SearchEngine::QueryBuilder;
     require Koha::SearchEngine::Search;
 
diff --git a/C4/Heading.pm b/C4/Heading.pm
index 7541a56205..8dede0c090 100644
--- a/C4/Heading.pm
+++ b/C4/Heading.pm
@@ -209,7 +209,7 @@ sub _search {
         push @value,    $self->{'search_form'};
     }
 
-    if ($self->{'thesaurus'}) {
+    if ($self->{'thesaurus'} && $ind2 ne ' ') {
         if ($ind2 eq '0') {
             $subject_heading_thesaurus = 'a';
         } elsif ($ind2 eq '1') {
@@ -224,8 +224,16 @@ sub _search {
             $subject_heading_thesaurus = 'k';
         } elsif ($ind2 eq '6') {
             $subject_heading_thesaurus = 'v';
-        } else {
+        } elsif ($ind2 eq '7') {
             $subject_heading_thesaurus = 'z';
+            push @marclist, 'thesaurus-conventions';
+            push @and_or, 'and';
+            push @excluding, '';
+            push @operator, 'is';
+            push @value, $self->{'thesaurus'};
+        } else {
+            # use lcsh for anything else
+            $subject_heading_thesaurus = 'a';
         }
         push @marclist, 'thesaurus';
         push @and_or, 'and';
@@ -234,14 +242,6 @@ sub _search {
         push @value, $subject_heading_thesaurus;
     }
 
-    if ($ind2 eq '7') {
-        push @marclist, 'thesaurus-conventions';
-        push @and_or, 'and';
-        push @excluding, '';
-        push @operator, 'is';
-        push @value, $self->{'thesaurus'};
-    }
-
     require Koha::SearchEngine::QueryBuilder;
     require Koha::SearchEngine::Search;
Comment 8 Janusz Kaczmarek 2023-03-02 12:57:28 UTC
(In reply to Frank Hansen from comment #6)
> (In reply to Janusz Kaczmarek from comment #3)
> > 
> > I would be curious to hear the opinions of those involved in the creation of
> > the bug fix 30280.
> 
> The main purpose of fix 30280 is to make Koha to distinguish authority
> records for subject headings (6XX fields) based on thesaurus. The fix is
> ​​not made for non-6XX fields.
> 
> However, as you mention, Koha hardcodes non-6XX fields to 'lcsh' (which is
> equivalent to subject-heading-thesaurus = 'a'). This means that all
> authority records must coded as lcsh. So yes, we need to do something about
> it. According to https://www.loc.gov/marc/bibliographic/bdx00.html indicator
> 2 is undefined for 1XX, 7XX and 8XX fields. I saw you changed it to undef
> which is probably fine, unless it has other side effects.
> 
> So, we only care for thesaurus value for the 6XX fields and skip filtering
> on based on thesaurus for non-6XX fields. I have chosen to make a slightly
> different solution for this. In C4::Heading_search I check if indicator 2
> has a value and is not blank. If it has a value, the filtering is performed
> on the thesaurus. But if it is empty, i.e. has a space, no filtering is
> performed. I made a fix for that, which I also mentioned in comment 58. I
> provide my fix as a comment so you can see how I made it. But it still
> doesn't solve the problem with Zebra and how it filters terms on the
> thesaurus.
> 
> My experience was that the thesaurus filtering in Zebra did not work after
> all, even though it was said to work. The same term from different thesauris
> were considered duplicates. So I wonder if something is broken here?
> 
> I really have no opinion on whether our thesaurus index should be called
> system-heading-thesaurus or system-heading-thesaurus-conventions. As long as
> it doesn't create confusion into thinking that the system-heading-thesaurus
> only contains 008/11, but also 040f.

OK, I have tried your new proposal (BTW, there are two versions in the comment https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32916#c7 ? I was able to apply only the first one).  I seems that it does not change much -- from my supplied test set only the three '[Ff]eminism' and 'Subject lcsh' are linked correctly, and only under ES.  Nothing under Zebra and also under Elastic the rest of headings remains unlinked.  So -- no visible change.

I fully understand the need to distinguish between thesauri while linking but we should not fix one thing and at the same time make other aspects not function.

Earlier, under Zebra, there was no functionality to distinguish the thesauri nor other aspects of authority 008 field while linking therefor it should be fixed as well.

And we cannot assume not force users to use 008/11 ='a' for authorities intended for biblio fields =! 6XX -- especially when 008/15 = 'b', 008/11 does not play any role.

Have you tested my second proposal?  Didn't it work for you?
Comment 9 Frank Hansen 2023-03-08 18:09:23 UTC
> 
> OK, I have tried your new proposal (BTW, there are two versions in the
> comment https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=32916#c7 ?
> I was able to apply only the first one).  I seems that it does not change
> much -- from my supplied test set only the three '[Ff]eminism' and 'Subject
> lcsh' are linked correctly, and only under ES.  Nothing under Zebra and also
> under Elastic the rest of headings remains unlinked.  So -- no visible
> change.
> 
> I fully understand the need to distinguish between thesauri while linking
> but we should not fix one thing and at the same time make other aspects not
> function.
> 
> Earlier, under Zebra, there was no functionality to distinguish the thesauri
> nor other aspects of authority 008 field while linking therefor it should be
> fixed as well.
> 
> And we cannot assume not force users to use 008/11 ='a' for authorities
> intended for biblio fields =! 6XX -- especially when 008/15 = 'b', 008/11
> does not play any role.
> 
> Have you tested my second proposal?  Didn't it work for you?

Yes, I tested the second proposal on my testing docker. After adding the syspref it worked in ElasticSearch. I also tried it with Zebra but unfortunately I didn't succeed. But when i ran link_bibs_to_authorities.pl I got those "oAuth error: Unsupported Use attribute (114) Subject-heading-thesaurus Bib-1". I haven't had time do dig deeper into that. I also have a kohadevbox available for testing. And I would like to test it on the kohadevbox as well.

Regards,
Frank Hansen
Comment 10 Nick Clemens 2023-03-08 18:16:45 UTC
The code here references a new pref, but doesn't add it

I have some proposed code on 33159 that I think will work.

I like the idea of hardcoding the mappings, the user has no real need to adjust these, linking just needs to work.

We should work together, I think I need a follow-up for notspecified, but let me know your thoughts
Comment 11 Janusz Kaczmarek 2023-03-08 19:15:19 UTC
(In reply to Nick Clemens from comment #10)
> The code here references a new pref, but doesn't add it

No, at the time there is no code to add the syspref - for the purpose of testing you would have to add it (i.e. LinkerStrictAuthInfo) as 'Local use' and set to 1 -- sorry for not having put it explicitly in test plan.

> 
> I have some proposed code on 33159 that I think will work.
> 

Yes, I saw it in the morning.  I want to test is as soon as I will able to. 

> I like the idea of hardcoding the mappings, the user has no real need to
> adjust these, linking just needs to work.
> 
> We should work together, I think I need a follow-up for notspecified, but
> let me know your thoughts

I will give you feedback after having tested your patch.  Meanwhile, you could test your proposal with my data set: https://bugs.koha-community.org/bugzilla3/attachment.cgi?id=146524 (which is a superset of Frank's test data). 

As a result, only one Feminism should stay unlinked (with Zebra and with ES):

650  4 $a Feminism

all the rest should be linked in both biblio records.
Comment 12 Frank Hansen 2023-03-09 15:58:32 UTC
(In reply to Nick Clemens from comment #10)
> The code here references a new pref, but doesn't add it
> 
> I have some proposed code on 33159 that I think will work.
> 
> I like the idea of hardcoding the mappings, the user has no real need to
> adjust these, linking just needs to work.
> 
> We should work together, I think I need a follow-up for notspecified, but
> let me know your thoughts

According to https://www.loc.gov/marc/bibliographic/bdx30.html, ind2 for non-6XX headers may  in many cases contain an empty value. But for 7XX headers, ind2 can either have blank (undefined) or the value '2' (analytic entry). Maybe that's something we should take into account? As is, all non-6XX headings will get 'lcsh' (or undef in Janusz proposal), which can be misleading in this case.
Comment 13 Nick Clemens 2023-03-22 13:04:43 UTC
Looking back over this I do think we ended up in very similar places. This is very good work, and I adopted some of it to complete 33277.

I favored my own code, as developers do sometimes, and didn't look as closely here once 33159 was PQA.

I think 33159 and 33277 can move forward together, but wanted to acknowledge that I ended up duplicating much of this work and give credit to Janusz
Comment 14 Phil Ringnalda 2023-05-04 00:12:25 UTC
Unsurprisingly, patches no longer apply.
Comment 15 Nick Clemens 2023-05-04 10:57:23 UTC

*** This bug has been marked as a duplicate of bug 33277 ***