Created attachment 131753 [details] [review]
Add a mapping for authority marc format field 040 to a new Subject-heading-thesaurus-conventions field

Current status: NEW




To test this patch do the following:

1. Load some sample authority records
./bulkmarcimport.pl -v -a -file /var/tmp/authorities.mrc -insert -c=MARC21

2. Load a bibliographic record
./bulkmarcimport.pl -v -b -file /var/tmp/biblios.mrc -insert -c=MARC21

3. Search the ISBN 0704328623 and verify that the subject heading is not linked

4. Apply the patch

5. Reset the ElasticSearch mappings (Administration > Search Engine Configuration (ElasticSearch))

6. Reindex ElasticSearch authorities index.
/home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -a -d -v

7. Link bibliographic records to authority records
/home/vagrant/kohaclone/misc/link_bibs_to_authorities.pl -v -l

8. Repeat the Search in step 3 and verify that all the Feminism headings are linked.

Created attachment 131754 [details] [review]
Handle subject headings from differens thesaurus

Created attachment 131755 [details]
Sample authority records

Created attachment 131756 [details]
Sample bibliographic record

Created attachment 131766 [details] [review]
Add a mapping for authority marc format field 040 to a new Subject-heading-thesaurus-conventions field

Removing tab chars

Created attachment 131767 [details] [review]
Handle subject headings from differens thesaurus

Created attachment 131768 [details] [review]
Removing tab character in Heading.pm

Created attachment 131769 [details] [review]
Bug 30280: Add a new mapping authority marc field 040$f

Created attachment 131770 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus

Created attachment 131771 [details] [review]
Bug 30280: Removing tab character in Heading.pm

Created attachment 131772 [details] [review]
Bug 30280: Add a new mapping authority marc field 040\$f

To test this patch do the following:

1. Load some sample authority records
./bulkmarcimport.pl -v -a -file /var/tmp/authorities.mrc -insert -c=MARC21

2. Load a bibliographic record
./bulkmarcimport.pl -v -b -file /var/tmp/biblios.mrc -insert -c=MARC21

3. Search the ISBN 0704328623 and verify that the subject heading is not linked

4. Apply the patch

5. Reset the ElasticSearch mappings (Administration > Search Engine Configuration (ElasticSearch))

    6. Reindex ElasticSearch authorities index.
    /home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -a -d -v

    7. Link bibliographic records to authority records
    /home/vagrant/kohaclone/misc/link_bibs_to_authorities.pl -v -l

    8. Repeat the Search in step 3 and verify that all the Feminism headings are linked.

Created attachment 131773 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus

Created attachment 131774 [details] [review]
Bug 30280: Removing tab character in Heading.pm

We are using multiple thesauruses as well and would like the linker to be able to distinguish between them too.  Here is a typical example:

600 1 0 ‡aGammage, Jeff.‡0n 2007005947‡9104714
650 _ 0 ‡aAdoptive parents‡vBiography.‡0sh 92003241 
650 _ 0 ‡aIntercountry adoption‡zChina.‡0sh 85067220 
650 _ 0 ‡aIntercountry adoption‡zUnited States.‡0sh2008104783‡9227784
650 _ 7 ‡aAdoptive parents.‡2homoit‡0homoit0000010‡9218151

We would like the linker to consider the 6XX$0 field and the 6XX$2 field when it selects the authority record for the 6XX$9 link.  The 6XX0 field matches with either the 010 (LC headings) or the 024 (other thesauruses).  Using the 6XX$0 to match would also make it possible for the linker to find the match for compound headings, which you can see it doesn't find for two of the 6XXs in my example.

Hi, (In reply to Sarah Cornell from comment #14)
> We are using multiple thesauruses as well and would like the linker to be
> able to distinguish between them too.  Here is a typical example:
> 
> 600 1 0 ‡aGammage, Jeff.‡0n 2007005947‡9104714
> 650 _ 0 ‡aAdoptive parents‡vBiography.‡0sh 92003241 
> 650 _ 0 ‡aIntercountry adoption‡zChina.‡0sh 85067220 
> 650 _ 0 ‡aIntercountry adoption‡zUnited States.‡0sh2008104783‡9227784
> 650 _ 7 ‡aAdoptive parents.‡2homoit‡0homoit0000010‡9218151
> 
> We would like the linker to consider the 6XX$0 field and the 6XX$2 field
> when it selects the authority record for the 6XX$9 link.  The 6XX0 field
> matches with either the 010 (LC headings) or the 024 (other thesauruses). 
> Using the 6XX$0 to match would also make it possible for the linker to find
> the match for compound headings, which you can see it doesn't find for two
> of the 6XXs in my example.

I understand your point. In Bug 17511 there is a discussion whether to move the AuthId link field from $9 to $0 or not. But there is a comment that $0 is a complex field. And the discussion seems to have died after that.

$0 is complex because it is repeatable and its content may have different meanings. It may even contain URI:s. We have such situation in our catalog as well. Here is an example:

=650  \7$aFeminism$0https://id.kb.se/term/sao/Feminism$2sao$0144892$94353
=650  \7$afeminism$2bnb$0144892$9216319
=650  \0$aFeminism$9216318

The first heading referring to the sao thesaurus. There are two occurences of the $0 subfield. The first one consist of a URI and the the second one is the ID of the authority record in the Royal Library catalog.

The second heading referring to the BNB thesaurus with one occurence of the $0 subfield which, i guess is the ID of the authority record in the BNB thesaurus system.

The last heading is our LC authority record.

When I viewed the code I couldn't see I understand your point. In Bug 17511 there is a discussion whether to move the AuthId link field from $9 to $0 or not. But there is a comment that $0 is complex. The discussion almost died after this comment. It is complex partly because subfield $0 is a repeatable field and may contain other ID:s with a different meaning and partly because it also may contain URI:s. We have such situation in our catalog as well. Here is an example taken from our catalog:

=650  \7$aFeminism$0https://id.kb.se/term/sao/Feminism$2sao$0144892$94353
=650  \7$afeminism$2bnb$0144892$9216319
=650  \0$aFeminism$9216318

The first heading referring to the sao thesaurus. There are two occurences of the $0 subfield. The first one consist of a URI and the the second one is the ID to the actual authority record at the Royal library.

The second heading referring to the BNB thesaurus with one occurence of the $0 subfield which probably is a ID from the BNB thesaurus system.

The last heading is our LC authority record.

I know that the LC field 010$az is an indexed field and can certainly be used to duplication control. But i'm not sure if it is doable to distinguish between LC records based on the 6XX$0 field.

Hi Frank.

I am getting an error message when resetting the mappings after the patch is applied (step 5 of the test plan) - this worked without error before the patch was applied:

DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st execute failed: Data truncated for column 'marc_type' at row 1 at /kohadevbox/koha/Koha/Objects.pm line 114
 at /usr/share/perl5/DBIx/Class/Exception.pm line 77

    in DBIx::Class::Exception::throw at /usr/share/perl5/DBIx/Class/Exception.pm line 77

       74:     my $self = { msg => $msg };
       75:     bless $self => $class;
       76: 
       77:     die $self;
       78: }
       79: 
       80: =head2 rethrow

   .....

David

Created attachment 137440 [details]
Screenshot - Example record before patch applied (subject terms have links)

Testing notes so far (using koha-testing-docker):

1. Enable Elasticsearch as the search engine and reindex (koha-elasticsearch --rebuild -d -b -a kohadev)

2. For step 1: misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21

3. For step 2: misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert -c=MARC21

4. For step 4, subject terms have search links (see attached screenshot) - search query example for one of the links http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:%22Feminism%22

5. For step 5, I get the error message in comment #16

(In reply to David Nind from comment #16)
> Hi Frank.
> 
> I am getting an error message when resetting the mappings after the patch is
> applied (step 5 of the test plan) - this worked without error before the
> patch was applied:
> 
> DBIx::Class::Storage::DBI::_dbh_execute(): DBI Exception: DBD::mysql::st
> execute failed: Data truncated for column 'marc_type' at row 1 at
> /kohadevbox/koha/Koha/Objects.pm line 114
>  at /usr/share/perl5/DBIx/Class/Exception.pm line 77
> 
>     in DBIx::Class::Exception::throw at
> /usr/share/perl5/DBIx/Class/Exception.pm line 77
> 
>        74:     my $self = { msg => $msg };
>        75:     bless $self => $class;
>        76: 
>        77:     die $self;
>        78: }
>        79: 
>        80: =head2 rethrow
> 
>    .....
> 
> David

Hi David,

I don't get this error. I don't using docker myself but a kohadevbox. I using the latest debian 11 box to create the koha enviroment with ElasticSearch 6. I did the following:

1. Creating the koha database instance by using the webinstaller.
2. Loaded demo records into the database (i using kshell when I work with the records)
perl ../misc4dev/insert_data.pl
rebuild_elasticsearch.pl -d -v
3. Enabling Elasticsearch as search engine in intranet
4. Loading the sample records with bulkmarkimport.pl:
./bulkmarcimport.pl -v -a -file /var/tmp/authorities.mrc -insert -c=MARC21
./bulkmarcimport.pl -v -b -file /var/tmp/biblios.mrc -insert -c=MARC21
5. I verified if the sample records were loaded properly. I experienced that the authority record was loaded and indexed ok. But the biblio record weren't indexed for some reason. They seems to have been loaded into the database but somehow Koha didn't indexed the record. This command worked for me before. I had to reindex the biblio records again before I applied the patch:
/home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -d -v
6. The sample biblio have topic terms but they are not linked to any authority records.
7. Applying the patch.
8. Resetting the mappings which work for me without any any errors as described above.
9. Rebuilding the authority index.
/home/vagrant/kohaclone/misc/search_tools/rebuild_elasticsearch.pl -a -d -v
10. Link bibliographic records to authority records
/home/vagrant/kohaclone/misc/link_bibs_to_authorities.pl -v -l
11. When I search the record, the topic_terms is linked.

Is there some differences between docker and kohadevbox when it comes to mappings?

Regards,
Frank

Hi Frank.

I think I'm getting the error because of lines 750-754 from the admin/searchengine/elasticsearch/mappings.yaml patch.

If I remove those lines, then I can reset the mappings without an error.

NORMARC was removed from Koha from 21.11 - see bug 187984

David

I tested (after making the change in comment #19) using koha-testing-docker.

Test plan:

1. Enable Elasticsearch as the search engine (SearchEngine system preference = Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).

2. Load sample authority records attached to the bug (in this case the file was saved in the koha directory - adjust file location as appropriate):[1]
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21

3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert -c=MARC21

4. Search for ISBN 0704328623 and verify that the subject terms are not linked to any authority records (if you hover over the link terms, all links should look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should have any ?q=an:XXX (where XXX = authority record ids) in the link).

5. Apply the patches.[2]

6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine configuration (Elasticsearch)).

7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

8. Link bibliographic records to authority records: misc/link_bibs_to_authorities.pl -v -l

9. Repeat the search in step 4 (or refresh the record details page) and verify that all the Feminism headings are linked to an authority record (should have ?q=an:XXX (where XXX = different authority record ids for the various feminism authority records)).[3]

[1] After importing the authorities, I got an error and had to reindex:
    
    misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21
    Characteristic MARC flavour: MARC21
    .One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
    .One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
    .One or more ElasticSearch errors occurred when indexing documents at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.

[2] After applying the patch I edited mappings.yaml to remove the nomarc entry that was added so the resetting the mappings worked
    vi admin/searchengine/elasticsearch/mappings.yaml -  remove lines 750-754 added by the mapping patch

[3] Not all links to feminism are linked to authority records. Links for subject terms for step 9 in order are:

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria | Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

(In reply to David Nind from comment #20)
> I tested (after making the change in comment #19) using koha-testing-docker.
> 
I finally manage to set up a koha-testing-docker myself. 

> Test plan:

> 
> [1] After importing the authorities, I got an error and had to reindex:
>     
>     misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc
> -insert -c=MARC21
>     Characteristic MARC flavour: MARC21
>     .One or more ElasticSearch errors occurred when indexing documents at
> /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
>     .One or more ElasticSearch errors occurred when indexing documents at
> /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
>     .One or more ElasticSearch errors occurred when indexing documents at
> /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 148.
I didn't got this error. Loading of authority record and bibliographic record worked fine for me.

> 
> [2] After applying the patch I edited mappings.yaml to remove the nomarc
> entry that was added so the resetting the mappings worked
>     vi admin/searchengine/elasticsearch/mappings.yaml -  remove lines
> 750-754 added by the mapping patch
Yes, normarc should be removed. 

> 
> [3] Not all links to feminism are linked to authority records. Links for
> subject terms for step 9 in order are:
> 
>     Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
> Litteratur
> 
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
>     http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"
> 
>     MARC info:
> 
>     650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Feminism
>       0 Authority record control number or standard number
> https://id.kb.se/term/sao/Feminism
>       2 Source of heading or term sao
>     650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element feminism
>       2 Source of heading or term bnb
>     650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Feminism
>     650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Feminism
>     650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Idéhistoria
>     650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
>       a Topical term or geographic name entry element Litteratur

This is actually correct, because the fourth Feminism term has second indicator 4 which means "Source not specified". So this term is only a local term and does not belonging to any specific thesaurus and thus not linked. To specify a source of a term, second indicator must be 7. The source of heading is then specified in subfield 2.

Created attachment 137818 [details] [review]
Bug 30280: Add a new mapping authority marc field 040\$f

Created attachment 137819 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus

Created attachment 137820 [details] [review]
Bug 30280: Removing tab character in Heading.pm

I fixed the mapping issue

Created attachment 137890 [details] [review]
Bug 30280: Add a new mapping authority marc field 040\$f

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137891 [details] [review]
Bug 30280: Handle subject headings from differens thesaurus

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137892 [details] [review]
Bug 30280: Removing tab character in Heading.pm

Signed-off-by: David Nind <david@davidnind.com>

(In reply to Frank Hansen from comment #25)
> I fixed the mapping issue

Thanks Frank!

I have re-tested and signed off.

Re replies on comment #21 - thanks for the explanations for my footnote [3]. I also didn't get the error message mentioned in my footnote [1] this time, so not sure what was going on whenI orginally tested this...

The QA Team may ask you to amend the patches as per the commit message guidelines https://wiki.koha-community.org/wiki/Commit_messages

I've had a go at the text for the release note, feel free to update it as I'm not sure I've described this correctly.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference = Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the authorities.mrc file was saved in the koha directory - adjust file location as appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert -c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert -c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked to any authority records (if you hover over the link terms, all links should look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records: misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify that the first three Feminism headings[1] are linked to an authority record (should have ?q=an:XXX (where XXX = different authority record IDs for the various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria | Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source not specified". So this term is only a local term and does not belong to any specific thesaurus and thus not linked. To specify a source of a term, the second indicator must be set as 7. The source of heading is then specified in subfield 2.

Created attachment 137897 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137898 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137899 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137900 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137901 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Created attachment 137902 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

(In reply to David Nind from comment #29)
> (In reply to Frank Hansen from comment #25)

Thanks David.

> > I fixed the mapping issue
> 
> The QA Team may ask you to amend the patches as per the commit message
> guidelines https://wiki.koha-community.org/wiki/Commit_messages
I have modified the commit messages a little bit. I used your Test plan. So I hope it looks better now. 

> 
> I've had a go at the text for the release note, feel free to update it as
> I'm not sure I've described this correctly.
I think the release notes well explain what the enhancement is about. I haven't changed anything it.

(In reply to Frank Hansen from comment #36)

Excellent! 

Good luck for QA.

Created attachment 139854 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 139855 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 139856 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Thx for the great work on the test plan and sample data, this makes it much easier to test :)

I am happy to add another sign-off, but for this to go through QA, we will also need unit tests to cover the changes in the .pm files. Please let us know if we can help with that!

You can switch back to signed off directly once the unit tests are done!

(In reply to Katrin Fischer from comment #41)
> Thx for the great work on the test plan and sample data, this makes it much
> easier to test :)
> 
> I am happy to add another sign-off, but for this to go through QA, we will
> also need unit tests to cover the changes in the .pm files. Please let us
> know if we can help with that!

Yes, there are already existing tests in these files:
t/db_dependent/Linker_Default.t
t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t 

They need some minor changes so they will PASS tests.

This test outputs warnings even if it says OK:
t/db_dependent/Linker_FirstMatch.t
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 38.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 79.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 80.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 81.
Use of uninitialized value $thesaurus in concatenation (.) or string at /kohadevbox/koha/C4/Linker/Default.pm line 82.
t/db_dependent/Linker_FirstMatch.t .. ok
All tests successful.

These warnings doesn't appear in t/db_dependent/Linker_Default.t 

I would like to add a few more test to t/db_dependent/Linker_Default.t to cover my changes in unit test.

Created attachment 140040 [details] [review]
30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140041 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140042 [details] [review]
Bug 30280 - Unit tests

Created attachment 140044 [details] [review]
Bug 30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140045 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140046 [details] [review]
Bug 30280: Add Unit tests

Sponsored-by: Lund University Library, Sweden

Created attachment 140284 [details] [review]
Bug 30280: Elasticsearch - Add 040f to Subject-heading-thesaurus-conventions (new) authority mapping index field (MARC21)

This patch adds 040 $f to a new field Subject-heading-thesaurus-conventions authority index mapping.

To test:
1) Apply patch
2) Reindex using rebuild_elasticsearch.pl -r

If you don't have access to a terminal (in a sandbox for example)
2a) Go to Administration > Search engine configuration, click "Reset mappings" and confirm
2b) Then reindex

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140285 [details] [review]
Bug 30280: Add support for subject headings from different thesaurus when using Elasticsearch

This patch allows subject headings from different thesaurus to co-exist in Koha.

Test plan:
1. Enable Elasticsearch as the search engine (SearchEngine system preference =
Elasticsearch) and reindex (koha-elasticsearch --rebuild -d -b -a kohadev).
2. Load sample authority records attached to the bug (in this case the
authorities.mrc file was saved in the koha directory - adjust file location as
appropriate):
   misc/migration_tools/bulkmarcimport.pl -v -a -file authorities.mrc -insert
-c=MARC21
3. Load the sample bibliographic record attached to the bug:
   misc/migration_tools/bulkmarcimport.pl -v -b -file biblios.mrc -insert
-c=MARC21
4. Search for ISBN 0704328623 and verify that the subject terms are not linked
to any authority records (if you hover over the link terms, all links should
look like ../cgi-bin/koha/catalogue/search.pl?q=su:"subjectterm" - none should
have any ?q=an:XXX (where XXX = authority record ids) in the link).
5. Apply the patches.
6. Reset the Elasticsearch mappings (Administration > Catalog > Search engine
configuration (Elasticsearch)).
7. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev
8. Link bibliographic records to authority records:
misc/link_bibs_to_authorities.pl -v -l
9. Repeat the search in step 4 (or refresh the record details page) and verify
that the first three Feminism headings[1] are linked to an authority record
(should have ?q=an:XXX (where XXX = different authority record IDs for the
various feminism authority records linked to a specific thesaurus)).
10. Reindex: koha-elasticsearch --rebuild -d -b -a kohadev

[1] Links for subject terms for step 9 in order are (the authority ids may be
different depending on how ou are testing):

    Subject(s:) Feminism | feminism | Feminism | Feminism | Idéhistoria |
Litteratur

    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1709
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1710
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=an:1708
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Feminism"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Idéhistoria"
    http://127.0.0.1:8081/cgi-bin/koha/catalogue/search.pl?q=su:"Litteratur"

    MARC info:

    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
      0 Authority record control number or standard number
https://id.kb.se/term/sao/Feminism
      2 Source of heading or term sao
    650 #7 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element feminism
      2 Source of heading or term bnb
    650 #0 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Feminism
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Idéhistoria
    650 #4 - SUBJECT ADDED ENTRY--TOPICAL TERM
      a Topical term or geographic name entry element Litteratur

   Note: The fourth Feminism term has second indicator 4 which means "Source
not specified". So this term is only a local term and does not belong to any
specific thesaurus and thus not linked. To specify a source of a term, the
second indicator must be set as 7. The source of heading is then specified in
subfield 2.

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140286 [details] [review]
Bug 30280: (follow-up) Removing tab character in Heading.pm

Sponsored-by: Lund University Library, Sweden

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Created attachment 140287 [details] [review]
Bug 30280: Add Unit tests

Sponsored-by: Lund University Library, Sweden

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>

Pushed to master for 22.11.

Nice work everyone, thanks!

hi Frank, could you maybe have a look at the new bugs in Blocks? bug 31924 and bug 31925.

(In reply to Katrin Fischer from comment #55)
> hi Frank, could you maybe have a look at the new bugs in Blocks? bug 31924
> and bug 31925.

Hi Katrin,

Well 31924 is probably a Zebra related issue. I got the impression that linking of subject headings based on thesaurus already works with Zebra. It is at least that's what I thought when I read this in the authority-koha-indexdefs.xml file:

  <kohaidx:index_subject_thesaurus tag="008" offset="11" length="1" detail_tag="040" detail_subfields="f">
    <kohaidx:target_index>Subject-heading-thesaurus:w</kohaidx:target_index>
  </kohaidx:index_subject_thesaurus>

Here is already a reference to 040$f which is a crucial part when it comes to other thesaurus.
 
But I see that the linking does not work at all in Zebra.

Here at Lund University, we are only using ElasticSearch. In fact, we Zebra is not running at all. So it is difficult for me to test Zebra in real life.

I'n not an expert on Zebra. Probably the new subject-heading-thesaurus-conventions index needs to be set up in Zebra as well. I'm not 100% certain how to do that. 

As for 31925, that's the next step. We don't use AutoLinkBiblios because it doesn't work. We import the subject headings from different thesaurus from Royal Library (Libris). So for avoid 30280 to being too complex, I have focused on getting the thesaurus based subject heading linking to work. Since AutoLinkBiblios only creates LCSH subject headings we cannot use this. But it would be awesome to get it to work later on.

The problem is, we broke things with this one for people and a lot of libraries - we need to find a fix.

(In reply to Katrin Fischer from comment #57)
> The problem is, we broke things with this one for people and a lot of
> libraries - we need to find a fix.

Katrin, I have been working on a fix to make it compatible with Zebra. There is some code to be uploaded as well. The patch should work in theory but I can get Zebra to work as expected. But it work nicely with Elasticsearch. Do you want me to submit the patch here or to Bug 31924? I will definitely need some help with setting up the search field correctly.

(In reply to Katrin Fischer from comment #57)
> The problem is, we broke things with this one for people and a lot of
> libraries - we need to find a fix.

I've had reports about linking problems (with Zebra), and it looks like this bug is the root cause. Very frustrating.

In theory Nick has fixed the problem via bug 33277 and bug 33557 although I haven't confirmed for sure yet.