Bug 30280 - Support authority records with common subject headings from different thesaurus
Summary: Support authority records with common subject headings from different thesaurus
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement with 1 vote (vote)
Assignee: Frank Hansen
QA Contact: Katrin Fischer
URL:
Keywords:
Depends on:
Blocks: 31925 31924 32916 33159 33277 33557
  Show dependency treegraph
 
Reported: 2022-03-11 15:41 UTC by Frank Hansen
Modified: 2023-09-18 03:39 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This enhancement adds support for displaying and linking to subject headings from different thesaurus when using Elasticsearch. The thesaurus used for the term is added to the authority record using 040$f. For the bibliographic record, set the second indicator to 7 for 650 and add the source to 650$2. For local terms, use 4 (Source not specified) as the indicator. Example for a bibliographic record: 650 _ 0 $a Feminism 650 _ 7 $a Feminism $2 sao 650 _ 7 $a Feminism $2 barn The first example above is the LCSH term. The other two terms are from sao (controlled Swedish subject heading system) and barn (Swedish children subject heading system). These three are using the same TOPIC_TERM Feminism, but they belong to different thesaurus.
Version(s) released in:
22.11.00


Attachments
Add a mapping for authority marc format field 040 to a new Subject-heading-thesaurus-conventions field (1.16 KB, patch)
2022-03-15 16:30 UTC, Frank Hansen
Details | Diff | Splinter Review
Handle subject headings from differens thesaurus (4.82 KB, patch)
2022-03-15 16:30 UTC, Frank Hansen
Details | Diff | Splinter Review
Sample authority records (412 bytes, application/x-gzip)
2022-03-15 16:32 UTC, Frank Hansen
Details
Sample bibliographic record (796 bytes, application/x-gzip)
2022-03-15 16:33 UTC, Frank Hansen
Details
Add a mapping for authority marc format field 040 to a new Subject-heading-thesaurus-conventions field (1.16 KB, patch)
2022-03-16 08:32 UTC, Frank Hansen
Details | Diff | Splinter Review
Handle subject headings from differens thesaurus (4.82 KB, patch)
2022-03-16 08:32 UTC, Frank Hansen
Details | Diff | Splinter Review
Removing tab character in Heading.pm (852 bytes, patch)
2022-03-16 08:32 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Add a new mapping authority marc field 040$f (1.06 KB, patch)
2022-03-16 09:09 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Handle subject headings from differens thesaurus (4.78 KB, patch)
2022-03-16 09:09 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Removing tab character in Heading.pm (809 bytes, patch)
2022-03-16 09:09 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Add a new mapping authority marc field 040\$f (1.06 KB, patch)
2022-03-16 09:29 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Handle subject headings from differens thesaurus (4.78 KB, patch)
2022-03-16 09:29 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Removing tab character in Heading.pm (809 bytes, patch)
2022-03-16 09:29 UTC, Frank Hansen
Details | Diff | Splinter Review
Screenshot - Example record before patch applied (subject terms have links) (56.20 KB, image/png)
2022-07-08 21:56 UTC, David Nind
Details
Bug 30280: Add a new mapping authority marc field 040\$f (962 bytes, patch)
2022-07-18 14:19 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Handle subject headings from differens thesaurus (5.47 KB, patch)
2022-07-18 14:19 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Removing tab character in Heading.pm (809 bytes, patch)
2022-07-18 14:20 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Add a new mapping authority marc field 040\$f (1010 bytes, patch)
2022-07-19 22:14 UTC, David Nind
Details | Diff | Splinter Review
Bug 30280: Handle subject headings from differens thesaurus (5.52 KB, patch)
2022-07-19 22:14 UTC, David Nind
Details | Diff | Splinter Review
Bug 30280: Removing tab character in Heading.pm (857 bytes, patch)
2022-07-19 22:15 UTC, David Nind
Details | Diff | Splinter Review
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21) (1.44 KB, patch)
2022-07-20 08:34 UTC, Frank Hansen
Details | Diff | Splinter Review
30280: Add support for subject headings from different thesaurus when using Elasticsearch (9.01 KB, patch)
2022-07-20 08:34 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: (follow-up) Removing tab character in Heading.pm (916 bytes, patch)
2022-07-20 08:34 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21) (1.44 KB, patch)
2022-07-20 08:36 UTC, Frank Hansen
Details | Diff | Splinter Review
30280: Add support for subject headings from different thesaurus when using Elasticsearch (9.01 KB, patch)
2022-07-20 08:36 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: (follow-up) Removing tab character in Heading.pm (916 bytes, patch)
2022-07-20 08:36 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21) (1.49 KB, patch)
2022-08-27 13:46 UTC, Katrin Fischer
Details | Diff | Splinter Review
30280: Add support for subject headings from different thesaurus when using Elasticsearch (9.07 KB, patch)
2022-08-27 13:46 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 30280: (follow-up) Removing tab character in Heading.pm (974 bytes, patch)
2022-08-27 13:47 UTC, Katrin Fischer
Details | Diff | Splinter Review
30280: Add support for subject headings from different thesaurus when using Elasticsearch (9.10 KB, patch)
2022-09-01 13:53 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: (follow-up) Removing tab character in Heading.pm (974 bytes, patch)
2022-09-01 13:53 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280 - Unit tests (5.25 KB, patch)
2022-09-01 13:58 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Add support for subject headings from different thesaurus when using Elasticsearch (9.10 KB, patch)
2022-09-01 14:13 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: (follow-up) Removing tab character in Heading.pm (974 bytes, patch)
2022-09-01 14:13 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Add Unit tests (5.29 KB, patch)
2022-09-01 14:13 UTC, Frank Hansen
Details | Diff | Splinter Review
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21) (1.49 KB, patch)
2022-09-07 13:25 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 30280: Add support for subject headings from different thesaurus when using Elasticsearch (9.10 KB, patch)
2022-09-07 13:25 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 30280: (follow-up) Removing tab character in Heading.pm (974 bytes, patch)
2022-09-07 13:25 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 30280: Add Unit tests (5.35 KB, patch)
2022-09-07 13:25 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Frank Hansen 2022-03-11 15:41:59 UTC
Consider this three following examples
650 _ 0 $a Feminism
650 _ 7 $a Feminism $2 sao
650 _ 7 $a Feminism $2 barn

The first example above is the LCSH term. The two other sao (controlled Swedish subject heading system) and barn (Swedish children subject heading system). These three using basically the same TOPIC_TERM Feminism but they belongs to different thesaurus. According to https://www.loc.gov/marc/authority/ (Library of Congress), subject heading thesaurus/system is specified in 008/11 in the authority record. When other thesaurus/system is used, the value of 008/11 is "z" (Other) and the convention used is marked up in field 040$f. But Koha currently not seems to support the use of different thesaurus. The script link_bibs_to_authorites.pl will treat authority records with the same TOPIC_TERM from different thesaurus as duplicates and will not be able to link the subject heading to its corresponding authority record. 

We would like if Koha could let authority records from different thesaurus co-exist in Koha and "learn" Koha how to distinguish between them.
Comment 1 Frank Hansen 2022-03-15 16:30:18 UTC
Created attachment 131753 [details] [review]
Add a mapping for authority marc format field 040 to a new Subject-heading-thesaurus-conventions field

Current status: NEW




To test this patch do the following:

1. Load some sample authority records
./bulkmarcimport.pl -v -a -file /var/tmp/authorities.mrc -insert -c=MARC21

2. Load a bibliographic record
./bulkmarcimport.pl -v -b -file /var/tmp/biblios.mrc -insert -c=MARC21

3. Search the ISBN 0704328623 and verify that the subject heading is not linked

4. Apply the patch

5. Reset the ElasticSearch mappings (Administration > Search Engine Configuration (ElasticSearch))

6. Reindex ElasticSearch authorities index.
/home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -a -d -v

7. Link bibliographic records to authority records
/home/vagrant/kohaclone/misc/link_bibs_to_authorities.pl -v -l

8. Repeat the Search in step 3 and verify that all the Feminism headings are linked.
Comment 2 Frank Hansen 2022-03-15 16:30:21 UTC
Created attachment 131754 [details] [review]
Handle subject headings from differens thesaurus
Comment 3 Frank Hansen 2022-03-15 16:32:36 UTC
Created attachment 131755 [details]
Sample authority records
Comment 4 Frank Hansen 2022-03-15 16:33:42 UTC
Created attachment 131756 [details]
Sample bibliographic record
Comment 5 Frank Hansen 2022-03-16 08:32:35 UTC
Created attachment 131766 [details] [review]
Add a mapping for authority marc format field 040 to a new Subject-heading-thesaurus-conventions field

Removing tab chars
Comment 6 Frank Hansen 2022-03-16 08:32:39 UTC
Created attachment 131767 [details] [review]
Handle subject headings from differens thesaurus
Comment 7 Frank Hansen 2022-03-16 08:32:42 UTC
Created attachment 131768 [details] [review]
Removing tab character in Heading.pm
Comment 8 Frank Hansen 2022-03-16 09:09:07 UTC
Created attachment 131769 [details] [review]
Bug 30280: Add a new mapping authority marc field 040$f
Comment 9 Frank Hansen 2022-03-16 09:09:11 UTC
Created attachment 131770 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus
Comment 10 Frank Hansen 2022-03-16 09:09:14 UTC
Created attachment 131771 [details] [review]
Bug 30280: Removing tab character in Heading.pm
Comment 11 Frank Hansen 2022-03-16 09:29:20 UTC
Created attachment 131772 [details] [review]
Bug 30280: Add a new mapping authority marc field 040\$f

To test this patch do the following:

1. Load some sample authority records
./bulkmarcimport.pl -v -a -file /var/tmp/authorities.mrc -insert -c=MARC21

2. Load a bibliographic record
./bulkmarcimport.pl -v -b -file /var/tmp/biblios.mrc -insert -c=MARC21

3. Search the ISBN 0704328623 and verify that the subject heading is not linked

4. Apply the patch

5. Reset the ElasticSearch mappings (Administration > Search Engine Configuration (ElasticSearch))

    6. Reindex ElasticSearch authorities index.
    /home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -a -d -v

    7. Link bibliographic records to authority records
    /home/vagrant/kohaclone/misc/link_bibs_to_authorities.pl -v -l

    8. Repeat the Search in step 3 and verify that all the Feminism headings are linked.
Comment 12 Frank Hansen 2022-03-16 09:29:24 UTC
Created attachment 131773 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus
Comment 13 Frank Hansen 2022-03-16 09:29:27 UTC
Created attachment 131774 [details] [review]
Bug 30280: Removing tab character in Heading.pm
Comment 14 Sarah Cornell 2022-07-07 15:17:37 UTC
We are using multiple thesauruses as well and would like the linker to be able to distinguish between them too.  Here is a typical example:

600 1 0 ‡aGammage, Jeff.‡0n 2007005947‡9104714
650 _ 0 ‡aAdoptive parents‡vBiography.‡0sh 92003241 
650 _ 0 ‡aIntercountry adoption‡zChina.‡0sh 85067220 
650 _ 0 ‡aIntercountry adoption‡zUnited States.‡0sh2008104783‡9227784
650 _ 7 ‡aAdoptive parents.‡2homoit‡0homoit0000010‡9218151

We would like the linker to consider the 6XX$0 field and the 6XX$2 field when it selects the authority record for the 6XX$9 link.  The 6XX0 field matches with either the 010 (LC headings) or the 024 (other thesauruses).  Using the 6XX$0 to match would also make it possible for the linker to find the match for compound headings, which you can see it doesn't find for two of the 6XXs in my example.
Comment 15 Frank Hansen 2022-07-08 14:50:54 UTC
Hi, (In reply to Sarah Cornell from comment #14)
> We are using multiple thesauruses as well and would like the linker to be
> able to distinguish between them too.  Here is a typical example:
> 
> 600 1 0 ‡aGammage, Jeff.‡0n 2007005947‡9104714
> 650 _ 0 ‡aAdoptive parents‡vBiography.‡0sh 92003241 
> 650 _ 0 ‡aIntercountry adoption‡zChina.‡0sh 85067220 
> 650 _ 0 ‡aIntercountry adoption‡zUnited States.‡0sh2008104783‡9227784
> 650 _ 7 ‡aAdoptive parents.‡2homoit‡0homoit0000010‡9218151
> 
> We would like the linker to consider the 6XX$0 field and the 6XX$2 field
> when it selects the authority record for the 6XX$9 link.  The 6XX0 field
> matches with either the 010 (LC headings) or the 024 (other thesauruses). 
> Using the 6XX$0 to match would also make it possible for the linker to find
> the match for compound headings, which you can see it doesn't find for two
> of the 6XXs in my example.

I understand your point. In Bug 17511 there is a discussion whether to move the AuthId link field from $9 to $0 or not. But there is a comment that $0 is a complex field. And the discussion seems to have died after that.

$0 is complex because it is repeatable and its content may have different meanings. It may even contain URI:s. We have such situation in our catalog as well. Here is an example:

=650  \7$aFeminism$0https://id.kb.se/term/sao/Feminism$2sao$0144892$94353
=650  \7$afeminism$2bnb$0144892$9216319
=650  \0$aFeminism$9216318

The first heading referring to the sao thesaurus. There are two occurences of the $0 subfield. The first one consist of a URI and the the second one is the ID of the authority record in the Royal Library catalog.

The second heading referring to the BNB thesaurus with one occurence of the $0 subfield which, i guess is the ID of the authority record in the BNB thesaurus system.

The last heading is our LC authority record.

When I viewed the code I couldn't see I understand your point. In Bug 17511 there is a discussion whether to move the AuthId link field from $9 to $0 or not. But there is a comment that $0 is complex. The discussion almost died after this comment. It is complex partly because subfield $0 is a repeatable field and may contain other ID:s with a different meaning and partly because it also may contain URI:s. We have such situation in our catalog as well. Here is an example taken from our catalog:

=650  \7$aFeminism$0https://id.kb.se/term/sao/Feminism$2sao$0144892$94353
=650  \7$afeminism$2bnb$0144892$9216319
=650  \0$aFeminism$9216318

The first heading referring to the sao thesaurus. There are two occurences of the $0 subfield. The first one consist of a URI and the the second one is the ID to the actual authority record at the Royal library.

The second heading referring to the BNB thesaurus with one occurence of the $0 subfield which probably is a ID from the BNB thesaurus system.

The last heading is our LC authority record.

I know that the LC field 010$az is an indexed field and can certainly be used to duplication control. But i'm not sure if it is doable to distinguish between LC records based on the 6XX$0 field.
Comment 16 David Nind 2022-07-08 21:52:44 UTC
Hi Frank.

I am getting an error message when resetting the mappings after the patch is applied (step 5 of the test plan) - this worked without error before the patch was applied:

DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Data truncated for column 'marc_type' at row 1 at /kohadevbox/koha/Koha/Objects.pm line 114
 at /usr/share/perl5/DBIx/Class/Exception.pm line 77

    in DBIx::Class::Exception::throw at /usr/share/perl5/DBIx/Class/Exception.pm line 77

       74:     my $self = { msg => $msg };
       75:     bless $self => $class;
       76: 
       77:     die $self;
       78: }
       79: 
       80: =head2 rethrow

   .....

David
Comment 17 David Nind 2022-07-08 21:56:04 UTC
Created attachment 137440 [details]
Screenshot - Example record before patch applied (subject terms have links)

Testing notes so far (using koha-testing-docker):

1. Enable Elasticsearch as the search engine and reindex (koha-elasticsearch --rebuild -d -b -a kohadev)

2. For step 1: misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21

3. For step 2: misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert -c=MARC21

4. For step 4, subject terms have search links (see attached screenshot) - search query example for one of the links http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:%22Feminism%22

5. For step 5, I get the error message in comment #16
Comment 18 Frank Hansen 2022-07-12 16:02:09 UTC
(In reply to David Nind from comment #16)
> Hi Frank.
> 
> I am getting an error message when resetting the mappings after the patch is
> applied (step 5 of the test plan) - this worked without error before the
> patch was applied:
> 
> DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st
> execute failed: Data truncated for column 'marc_type' at row 1 at
> /kohadevbox/koha/Koha/Objects.pm line 114
>  at /usr/share/perl5/DBIx/Class/Exception.pm line 77
> 
>     in DBIx::Class::Exception::throw at
> /usr/share/perl5/DBIx/Class/Exception.pm line 77
> 
>        74:     my $self = { msg => $msg };
>        75:     bless $self => $class;
>        76: 
>        77:     die $self;
>        78: }
>        79: 
>        80: =head2 rethrow
> 
>    .....
> 
> David

Hi David,

I don't get this error. I don't using docker myself but a kohadevbox. I using the latest debian 11 box to create the koha enviroment with ElasticSearch 6. I did the following:

1. Creating the koha database instance by using the webinstaller.
2. Loaded demo records into the database (i using kshell when I work with the records)
perl ../misc4dev/insert_data.pl
rebuild_elasticsearch.pl -d -v
3. Enabling Elasticsearch as search engine in intranet
4. Loading the sample records with bulkmarkimport.pl:
./bulkmarcimport.pl -v -a -file /var/tmp/authorities.mrc -insert -c=MARC21
./bulkmarcimport.pl -v -b -file /var/tmp/biblios.mrc -insert -c=MARC21
5. I verified if the sample records were loaded properly. I experienced that the authority record was loaded and indexed ok. But the biblio record weren't indexed for some reason. They seems to have been loaded into the database but somehow Koha didn't indexed the record. This command worked for me before. I had to reindex the biblio records again before I applied the patch:
/home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -d -v
6. The sample biblio have topic terms but they are not linked to any authority records.
7. Applying the patch.
8. Resetting the mappings which work for me without any any errors as described above.
9. Rebuilding the authority index.
/home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -a -d -v
10. Link bibliographic records to authority records
/home/vagrant/kohaclone/misc/link_bibs_to_authorities.pl -v -l
11. When I search the record, the topic_terms is linked.

Is there some differences between docker and kohadevbox when it comes to mappings?

Regards,
Frank
Comment 19 David Nind 2022-07-13 22:38:48 UTC
Hi Frank.

I think I'm getting the error because of lines 750-754 from the admin/searchengine/elasticsearch/mappings.yaml patch.

If I remove those lines, then I can reset the mappings without an error.

NORMARC was removed from Koha from 21.11 - see bug 187984

David
Comment 20 David Nind 2022-07-13 23:41:06 UTC
I tested (after making the change in comment #19) using koha-testing-docker.

Test plan:

1. Enable Elasticsearch as the search engine (SearchEngine system preference = Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).

2. Load sample authority records attached to the bug (in this case the file was saved in the koha directory - adjust file location as appropriate):[1]
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21

3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert -c=MARC21

4. Search for ISBN 0704328623 and verify that the subject terms are not linked to any authority records (if you hover over the link terms, all links should look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should have any ?q=an:XXX (where XXX = authority record ids) in the link).

5. Apply the patches.[2]

6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine configuration (Elasticsearch)).

7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

8. Link bibliographic records to authority records: misc/link_bibs_to_authorities.pl -v -l

9. Repeat the search in step 4 (or refresh the record details page) and verify that all the Feminism headings are linked to an authority record (should have ?q=an:XXX (where XXX = different authority record ids for the various feminism authority records)).[3]

[1] After importing the authorities, I got an error and had to reindex:
    
    misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21
    Characteristic MARC flavour: MARC21
    .One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
    .One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
    .One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.

[2] After applying the patch I edited mappings.yaml to remove the nomarc entry that was added so the resetting the mappings worked
    vi admin/searchengine/elasticsearch/mappings.yaml -  remove lines 750-754 added by the mapping patch

[3] Not all links to feminism are linked to authority records. Links for subject terms for step 9 in order are:

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria | Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur
Comment 21 Frank Hansen 2022-07-18 12:48:39 UTC
(In reply to David Nind from comment #20)
> I tested (after making the change in comment #19) using koha-testing-docker.
> 
I finally manage to set up a koha-testing-docker myself. 

> Test plan:

> 
> [1] After importing the authorities, I got an error and had to reindex:
>     
>     misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc
> -insert -c=MARC21
>     Characteristic MARC flavour: MARC21
>     .One or more ElasticSearch errors occurred when indexing documents at
> /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
>     .One or more ElasticSearch errors occurred when indexing documents at
> /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
>     .One or more ElasticSearch errors occurred when indexing documents at
> /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
I didn't got this error. Loading of authority record and bibliographic record worked fine for me.

> 
> [2] After applying the patch I edited mappings.yaml to remove the nomarc
> entry that was added so the resetting the mappings worked
>     vi admin/searchengine/elasticsearch/mappings.yaml -  remove lines
> 750-754 added by the mapping patch
Yes, normarc should be removed. 

> 
> [3] Not all links to feminism are linked to authority records. Links for
> subject terms for step 9 in order are:
> 
>     Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
> Litteratur
> 
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"
> 
>     MARC info:
> 
>     650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Feminism
>       0 Authority record control number or standard number
> https://id.kb.se/term/sao/Feminism
>       2 Source of heading or term sao
>     650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element feminism
>       2 Source of heading or term bnb
>     650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Feminism
>     650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Feminism
>     650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Idéhistoria
>     650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Litteratur

This is actually correct, because the fourth Feminism term has second indicator 4 which means "Source not specified". So this term is only a local term and does not belonging to any specific thesaurus and thus not linked. To specify a source of a term, second indicator must be 7. The source of heading is then specified in subfield 2.
Comment 22 Frank Hansen 2022-07-18 14:19:54 UTC
Created attachment 137818 [details] [review]
Bug 30280: Add a new mapping authority marc field 040\$f
Comment 23 Frank Hansen 2022-07-18 14:19:59 UTC
Created attachment 137819 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus
Comment 24 Frank Hansen 2022-07-18 14:20:05 UTC
Created attachment 137820 [details] [review]
Bug 30280: Removing tab character in Heading.pm
Comment 25 Frank Hansen 2022-07-18 14:26:30 UTC
I fixed the mapping issue
Comment 26 David Nind 2022-07-19 22:14:50 UTC
Created attachment 137890 [details] [review]
Bug 30280: Add a new mapping authority marc field 040\$f

Signed-off-by: David Nind <david@davidnind.com>
Comment 27 David Nind 2022-07-19 22:14:55 UTC
Created attachment 137891 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus

Signed-off-by: David Nind <david@davidnind.com>
Comment 28 David Nind 2022-07-19 22:15:01 UTC
Created attachment 137892 [details] [review]
Bug 30280: Removing tab character in Heading.pm

Signed-off-by: David Nind <david@davidnind.com>
Comment 29 David Nind 2022-07-19 22:56:28 UTC
(In reply to Frank Hansen from comment #25)
> I fixed the mapping issue

Thanks Frank!

I have re-tested and signed off.

Re replies on comment #21 - thanks for the explanations for my footnote [3]. I also didn't get the error message mentioned in my footnote [1] this time, so not sure what was going on whenI orginally tested this...

The QA Team may ask you to amend the patches as per the commit message guidelines https://wiki.koha-community.org/wiki/Commit_messages

I've had a go at the text for the release note, feel free to update it as I'm not sure I've described this correctly.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference = Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the authorities.mrc file was saved in the koha directory - adjust file location as appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert -c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked to any authority records (if you hover over the link terms, all links should look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records: misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify that the first three Feminism headings[1] are linked to an authority record (should have ?q=an:XXX (where XXX = different authority record IDs for the various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria | Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source not specified". So this term is only a local term and does not belong to any specific thesaurus and thus not linked. To specify a source of a term, the second indicator must be set as 7. The source of heading is then specified in subfield 2.
Comment 30 Frank Hansen 2022-07-20 08:34:39 UTC
Created attachment 137897 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>
Comment 31 Frank Hansen 2022-07-20 08:34:43 UTC
Created attachment 137898 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>
Comment 32 Frank Hansen 2022-07-20 08:34:48 UTC
Created attachment 137899 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>
Comment 33 Frank Hansen 2022-07-20 08:36:30 UTC
Created attachment 137900 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>
Comment 34 Frank Hansen 2022-07-20 08:36:35 UTC
Created attachment 137901 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>
Comment 35 Frank Hansen 2022-07-20 08:36:39 UTC
Created attachment 137902 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>
Comment 36 Frank Hansen 2022-07-20 08:52:00 UTC
(In reply to David Nind from comment #29)
> (In reply to Frank Hansen from comment #25)

Thanks David.

> > I fixed the mapping issue
> 
> The QA Team may ask you to amend the patches as per the commit message
> guidelines https://wiki.koha-community.org/wiki/Commit_messages
I have modified the commit messages a little bit. I used your Test plan. So I hope it looks better now. 

> 
> I've had a go at the text for the release note, feel free to update it as
> I'm not sure I've described this correctly.
I think the release notes well explain what the enhancement is about. I haven't changed anything it.
Comment 37 David Nind 2022-07-20 08:56:49 UTC
(In reply to Frank Hansen from comment #36)

Excellent! 

Good luck for QA.
Comment 38 Katrin Fischer 2022-08-27 13:46:51 UTC
Created attachment 139854 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 39 Katrin Fischer 2022-08-27 13:46:57 UTC
Created attachment 139855 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 40 Katrin Fischer 2022-08-27 13:47:03 UTC
Created attachment 139856 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 41 Katrin Fischer 2022-08-27 13:48:59 UTC
Thx for the great work on the test plan and sample data, this makes it much easier to test :)

I am happy to add another sign-off, but for this to go through QA, we will also need unit tests to cover the changes in the .pm files. Please let us know if we can help with that!
Comment 42 Katrin Fischer 2022-08-27 13:49:41 UTC
You can switch back to signed off directly once the unit tests are done!
Comment 43 Frank Hansen 2022-08-31 12:00:39 UTC
(In reply to Katrin Fischer from comment #41)
> Thx for the great work on the test plan and sample data, this makes it much
> easier to test :)
> 
> I am happy to add another sign-off, but for this to go through QA, we will
> also need unit tests to cover the changes in the .pm files. Please let us
> know if we can help with that!

Yes, there are already existing tests in these files:
t/db_dependent/Linker_Default.t
t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t 

They need some minor changes so they will PASS tests.

This test outputs warnings even if it says OK:
t/db_dependent/Linker_FirstMatch.t
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 38.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 79.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 80.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 81.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 82.
t/db_dependent/Linker_FirstMatch.t .. ok
All tests successful.

These warnings doesn't appear in t/db_dependent/Linker_Default.t 

I would like to add a few more test to t/db_dependent/Linker_Default.t to cover my changes in unit test.
Comment 44 Frank Hansen 2022-09-01 13:53:49 UTC
Created attachment 140040 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 45 Frank Hansen 2022-09-01 13:53:54 UTC
Created attachment 140041 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 46 Frank Hansen 2022-09-01 13:58:51 UTC
Created attachment 140042 [details] [review]
Bug 30280 - Unit tests
Comment 47 Frank Hansen 2022-09-01 14:13:46 UTC
Created attachment 140044 [details] [review]
Bug 30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 48 Frank Hansen 2022-09-01 14:13:51 UTC
Created attachment 140045 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 49 Frank Hansen 2022-09-01 14:13:55 UTC
Created attachment 140046 [details] [review]
Bug 30280: Add Unit tests

Sponsored-by: Lund University Library, Sweden
Comment 50 Katrin Fischer 2022-09-07 13:25:34 UTC
Created attachment 140284 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 51 Katrin Fischer 2022-09-07 13:25:41 UTC
Created attachment 140285 [details] [review]
Bug 30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 52 Katrin Fischer 2022-09-07 13:25:45 UTC
Created attachment 140286 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 53 Katrin Fischer 2022-09-07 13:25:51 UTC
Created attachment 140287 [details] [review]
Bug 30280: Add Unit tests

Sponsored-by: Lund University Library, Sweden

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 54 Tomás Cohen Arazi 2022-10-03 13:10:00 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 55 Katrin Fischer 2022-10-22 09:23:39 UTC
hi Frank, could you maybe have a look at the new bugs in Blocks? bug 31924 and bug 31925.
Comment 56 Frank Hansen 2022-10-25 15:28:50 UTC
(In reply to Katrin Fischer from comment #55)
> hi Frank, could you maybe have a look at the new bugs in Blocks? bug 31924
> and bug 31925.

Hi Katrin,

Well 31924 is probably a Zebra related issue. I got the impression that linking of subject headings based on thesaurus already works with Zebra. It is at least that's what I thought when I read this in the authority-koha-indexdefs.xml file:

  <kohaidx:index_subject_thesaurus tag="008" offset="11" length="1" detail_tag="040" detail_subfields="f">
    <kohaidx:target_index>Subject-heading-thesaurus:w</kohaidx:target_index>
  </kohaidx:index_subject_thesaurus>

Here is already a reference to 040$f which is a crucial part when it comes to other thesaurus.
 
But I see that the linking does not work at all in Zebra.

Here at Lund University, we are only using ElasticSearch. In fact, we Zebra is not running at all. So it is difficult for me to test Zebra in real life.

I'n not an expert on Zebra. Probably the new subject-heading-thesaurus-conventions index needs to be set up in Zebra as well. I'm not 100% certain how to do that. 

As for 31925, that's the next step. We don't use AutoLinkBiblios because it doesn't work. We import the subject headings from different thesaurus from Royal Library (Libris). So for avoid 30280 to being too complex, I have focused on getting the thesaurus based subject heading linking to work. Since AutoLinkBiblios only creates LCSH subject headings we cannot use this. But it would be awesome to get it to work later on.
Comment 57 Katrin Fischer 2022-11-17 13:03:04 UTC
The problem is, we broke things with this one for people and a lot of libraries - we need to find a fix.
Comment 58 Frank Hansen 2022-11-25 19:19:56 UTC
(In reply to Katrin Fischer from comment #57)
> The problem is, we broke things with this one for people and a lot of
> libraries - we need to find a fix.

Katrin, I have been working on a fix to make it compatible with Zebra. There is some code to be uploaded as well. The patch should work in theory but I can get Zebra to work as expected. But it work nicely with Elasticsearch. Do you want me to submit the patch here or to Bug 31924? I will definitely need some help with setting up the search field correctly.
Comment 59 David Cook 2023-09-18 03:39:43 UTC
(In reply to Katrin Fischer from comment #57)
> The problem is, we broke things with this one for people and a lot of
> libraries - we need to find a fix.

I've had reports about linking problems (with Zebra), and it looks like this bug is the root cause. Very frustrating.

In theory Nick has fixed the problem via bug 33277 and bug 33557 although I haven't confirmed for sure yet.