Bug 18017

Summary: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration
Product: Koha Reporter: Alex Arnaud <alex.arnaud>
Component: CatalogingAssignee: Fridolin Somers <fridolin.somers>
Status: CLOSED FIXED QA Contact: Victor Grousset/tuxayo <victor>
Severity: major    
Priority: P5 - low CC: andrewfh, carnold, dcook, didier.gautheron, fridolin.somers, jonathan.druart, laurence.rault, m.de.rooy, martin.renvoize, victor
Version: unspecified   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27299
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=29191
Change sponsored?: --- Patch complexity: Small patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.05.00,20.11.06
Bug Depends on: 27299    
Bug Blocks: 28908    
Attachments: Bug 18017 - add index_matching_heading to create indexes for matching an entire authority heading
Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration
Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration
Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration

Description Alex Arnaud 2017-01-31 09:35:14 UTC
This is a unimarc bug only:

When importing biblios from Z39.50, related authorities are wrong matched by the linker. This causes duplicates of authorities. The problem come from a bad marc definition in unimarc/authorities/authority-koha-indexdefs.xml which does not create a index for matching an entire heading.
Comment 1 Alex Arnaud 2017-01-31 09:38:52 UTC
Created attachment 59675 [details] [review]
Bug 18017 - add index_matching_heading to create indexes for matching an entire authority heading
Comment 2 Jonathan Druart 2017-02-16 13:21:21 UTC
A test plan to help testers?
Comment 3 Laurence Rault 2017-07-28 14:58:50 UTC
If I understand the patch, a test plan could be

Test plan

BiblioAddsAuthorities ON
AutoCreateAuthorities ON

1) Import Z3950 record and save it : authorities are created
2) Import the same record again, when saving say it is not a duplicate : 
Without the patch : new authorities (duplicates) are created for the same values
With the patch : new record should be linked with existing authorities



Tested on sandbox : 

BiblioAddsAuthorities ON
AutoCreateAuthorities ON

1) Import Z3950 record and save it : authorities are created
2) Import the same record again, when saving say it is not a duplicate : 
Without the patch : new authorities (duplicates) are created for the same values
With the patch : new authorities (duplicates) are created for the same values : KO
Comment 4 Alex Arnaud 2017-11-17 11:13:46 UTC
Full test plan is:

BiblioAddsAuthorities ON
AutoCreateAuthorities ON

1) replace /koha_conf_dir/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
   with /koha_clone/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
2) replace /koha_conf_dir/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
   with /koha_clone/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
3) reindex authorities,
4) Import Z3950 record and save it : authorities are created
5) Import the same record again, when saving say it is not a duplicate : 
Without the patch : new authorities (duplicates) are created for the same values
With the patch : new record should be linked with existing authorities
Comment 5 Victor Grousset/tuxayo 2017-11-24 14:46:47 UTC
After applying the patch, when reimporting, the authority of the main author is still duplicated. (different id on the "Main Author" link)

I even re-indexed authorities after applying the patch.
./src/misc/migration_tools/rebuild_zebra.pl -r -a -v
Comment 6 Victor Grousset/tuxayo 2018-05-11 07:05:55 UTC
Retried:

1) Set sysprefs
   BiblioAddsAuthorities ON
   AutoCreateAuthorities ON
2) Replace xsl files
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
3) reindex authorities
     ./misc/migration_tools/rebuild_zebra.pl -r -a -v
4) Import Z3950 record and save it : authorities are created
     go to staff:/cgi-bin/koha/cataloguing/addbooks.pl
5) Reimport the same record (when asked, say that it's not a duplicate)
6) The authority should have been duplicated (different url)
7) Apply this patch
8) Replace again the xsl files
9) reindex authorities
10) Reimport the same record
11) The authority should have not been duplicated
      But for me it was.
Comment 7 Cori Lynn Arnold 2018-10-17 13:38:37 UTC
Followed testing steps, but authority is duplicated.

Added book "Sometimes I Lie" twice, and got two "Feeney, Alice, author" records in authorities.
Comment 8 Didier Gautheron 2020-11-25 08:42:57 UTC
Hi

I can only reproduce https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18017#c7 on MARC21 and it's solved by 21958, but this patch is about UNIMARC and it's working for me, so back to needs Signoff.

Regards
Didier
Comment 9 Victor Grousset/tuxayo 2020-12-21 21:13:10 UTC
As it's an UNIMARC bug and it's getting old, I think it's okay to have a signoff from the same entity, so Didier or someone else at BibLibre, you can sign this off.
Comment 10 Fridolin Somers 2020-12-22 09:17:21 UTC
I'm on creating more patches so I set to ASSIGNED
Comment 11 Fridolin Somers 2020-12-22 10:33:46 UTC
Created attachment 114588 [details] [review]
Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration

Like for MARC21, UNIMARC authorities has subdivisions form, general,
chronological and geographic.

In C4::Heading::UNIMARC, use subdivisions in _get_search_heading like in C4::Heading::MARC21.

Adds subdivisions variables into UNIMARC authorities zebra configuration.

Note that unlike MARC21 geographic is subfield $y and chronological is subfield $z.
See https://www.ifla.org/publications/unimarc-formats-and-related-documentation
Comment 12 Fridolin Somers 2020-12-22 10:34:08 UTC
Created attachment 114589 [details] [review]
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration

For a good management of autorities linking to biblio records,
MARC21 uses index_heading and index_match_heading in authorities zebra configuration.
UNIMARC configuration must use the same.

This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading
in order to be maximum close to MARC21 authorities zebra configuration.
See changes made in MARC21 :
https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896

It fixes some indexes names : Personal-name-see => Personal-name-see-from

Removes useless Term-geographic index, a duplicate of Name-geographic.

Sometimes parallel 7xx form whas only on $a, it must contains same subfields
has the main heading.

Test plan :
===========
1.0) Use a UNIMARC install without patch
1.1) Set sysprefs
     BiblioAddsAuthorities = ON
     AutoCreateAuthorities = ON
     LinkerModule = First Match
1.2) Replace authorities zebra configuration files
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
1.3) Restart zebra server and indexer services
1.4) Reindex authorities
     ./misc/migration_tools/rebuild_zebra.pl -r -a -v
1.5) Search in Z3950 a record with complex heading (with subdivisions),
     for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues"
1.6) Import this record and save it : authorities are created
     go to staff:/cgi-bin/koha/cataloguing/addbooks.pl
1.7) Reimport the same record (when asked, say that it's not a duplicate)
1.8) The authority should have been duplicated :
     different url and different $9 value
2.0) Apply this patch
2.1) Replace again the authorities zebra configuration files
2.2) Restart zebra server and indexer services
2.3) Reindex authorities
2.4) Reimport the same record
2.5) The authority should have not been duplicated
3.0) Play with authorities search to check every mode :
     Search main heading ($a only)
     Search main heading
     Search all headings
     Search entire record
Comment 13 Fridolin Somers 2020-12-22 10:49:43 UTC
Created attachment 114590 [details] [review]
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration

For a good management of autorities linking to biblio records,
MARC21 uses index_heading and index_match_heading in authorities zebra configuration.
UNIMARC configuration must use the same.

This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading
in order to be maximum close to MARC21 authorities zebra configuration.
See changes made in MARC21 :
https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896

It fixes some indexes names : Personal-name-see => Personal-name-see-from

Removes useless Term-geographic index, a duplicate of Name-geographic.

Sometimes parallel 7xx form whas only on $a, it must contains same subfields
has the main heading.

Test plan :
===========
1.0) Use a UNIMARC install without patch
1.1) Set sysprefs
     BiblioAddsAuthorities = ON
     AutoCreateAuthorities = ON
     LinkerModule = First Match
1.2) Replace authorities zebra configuration files
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
1.3) Restart zebra server and indexer services
1.4) Reindex authorities
     ./misc/migration_tools/rebuild_zebra.pl -r -a -v
1.5) Search in Z3950 a record with complex heading (with subdivisions),
     for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues"
1.6) Import this record and save it : authorities are created
     go to staff:/cgi-bin/koha/cataloguing/addbooks.pl
1.7) Reimport the same record (when asked, say that it's not a duplicate)
1.8) The authority should have been duplicated :
     different url and different $9 value
2.0) Apply this patch
2.1) Replace again the authorities zebra configuration files
2.2) Restart zebra server and indexer services
2.3) Reindex authorities
2.4) Reimport the same record
2.5) The authority should have not been duplicated
3.0) Play with authorities search to check every mode :
     Search main heading ($a only)
     Search main heading
     Search all headings
     Search entire record
Comment 14 Fridolin Somers 2020-12-22 11:10:22 UTC
See also https://github.com/indexdata/idzebra/issues/24
Comment 15 Martin Renvoize 2021-02-23 09:34:38 UTC
Created attachment 117182 [details] [review]
Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration

Like for MARC21, UNIMARC authorities has subdivisions form, general,
chronological and geographic.

In C4::Heading::UNIMARC, use subdivisions in _get_search_heading like in C4::Heading::MARC21.

Adds subdivisions variables into UNIMARC authorities zebra configuration.

Note that unlike MARC21 geographic is subfield $y and chronological is subfield $z.
See https://www.ifla.org/publications/unimarc-formats-and-related-documentation

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 16 Martin Renvoize 2021-02-23 09:34:43 UTC
Created attachment 117183 [details] [review]
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration

For a good management of autorities linking to biblio records,
MARC21 uses index_heading and index_match_heading in authorities zebra configuration.
UNIMARC configuration must use the same.

This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading
in order to be maximum close to MARC21 authorities zebra configuration.
See changes made in MARC21 :
https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896

It fixes some indexes names : Personal-name-see => Personal-name-see-from

Removes useless Term-geographic index, a duplicate of Name-geographic.

Sometimes parallel 7xx form whas only on $a, it must contains same subfields
has the main heading.

Test plan :
===========
1.0) Use a UNIMARC install without patch
1.1) Set sysprefs
     BiblioAddsAuthorities = ON
     AutoCreateAuthorities = ON
     LinkerModule = First Match
1.2) Replace authorities zebra configuration files
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
1.3) Restart zebra server and indexer services
1.4) Reindex authorities
     ./misc/migration_tools/rebuild_zebra.pl -r -a -v
1.5) Search in Z3950 a record with complex heading (with subdivisions),
     for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues"
1.6) Import this record and save it : authorities are created
     go to staff:/cgi-bin/koha/cataloguing/addbooks.pl
1.7) Reimport the same record (when asked, say that it's not a duplicate)
1.8) The authority should have been duplicated :
     different url and different $9 value
2.0) Apply this patch
2.1) Replace again the authorities zebra configuration files
2.2) Restart zebra server and indexer services
2.3) Reindex authorities
2.4) Reimport the same record
2.5) The authority should have not been duplicated
3.0) Play with authorities search to check every mode :
     Search main heading ($a only)
     Search main heading
     Search all headings
     Search entire record

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 17 Martin Renvoize 2021-02-23 09:36:36 UTC
OK, I'm pretty familiar with the code around this in MARC21 and it looks like a reasonably good representation for UNIMARC here.. I did some testing to the extent I understand UNIMARC and it all works well..

Signing off..

I'll request Victor to take a look from a QA standpoint, hopefully he'll spot any UNIMARC issues I may not and can get it over the finish line.
Comment 18 Victor Grousset/tuxayo 2021-02-25 17:20:24 UTC
Created attachment 117350 [details] [review]
Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration

Like for MARC21, UNIMARC authorities has subdivisions form, general,
chronological and geographic.

In C4::Heading::UNIMARC, use subdivisions in _get_search_heading like in C4::Heading::MARC21.

Adds subdivisions variables into UNIMARC authorities zebra configuration.

Note that unlike MARC21 geographic is subfield $y and chronological is subfield $z.
See https://www.ifla.org/publications/unimarc-formats-and-related-documentation

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 19 Victor Grousset/tuxayo 2021-02-25 17:20:28 UTC
Created attachment 117351 [details] [review]
Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration

For a good management of autorities linking to biblio records,
MARC21 uses index_heading and index_match_heading in authorities zebra configuration.
UNIMARC configuration must use the same.

This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading
in order to be maximum close to MARC21 authorities zebra configuration.
See changes made in MARC21 :
https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896

It fixes some indexes names : Personal-name-see => Personal-name-see-from

Removes useless Term-geographic index, a duplicate of Name-geographic.

Sometimes parallel 7xx form whas only on $a, it must contains same subfields
has the main heading.

Test plan :
===========
1.0) Use a UNIMARC install without patch
1.1) Set sysprefs
     BiblioAddsAuthorities = ON
     AutoCreateAuthorities = ON
     LinkerModule = First Match
1.2) Replace authorities zebra configuration files
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml
     cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl
1.3) Restart zebra server and indexer services
1.4) Reindex authorities
     ./misc/migration_tools/rebuild_zebra.pl -r -a -v
1.5) Search in Z3950 a record with complex heading (with subdivisions),
     for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues"
1.6) Import this record and save it : authorities are created
     go to staff:/cgi-bin/koha/cataloguing/addbooks.pl
1.7) Reimport the same record (when asked, say that it's not a duplicate)
1.8) The authority should have been duplicated :
     different url and different $9 value
2.0) Apply this patch
2.1) Replace again the authorities zebra configuration files
2.2) Restart zebra server and indexer services
2.3) Reindex authorities
2.4) Reimport the same record
2.5) The authority should have not been duplicated. Compare with both
       existing records to see which the 3rd has been matched against.
3.0) Play with authorities search to check every mode :
     Search main heading ($a only)
     Search main heading
     Search all headings
     Search entire record

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 20 Victor Grousset/tuxayo 2021-02-25 17:23:39 UTC
Finally I have an UNIMARC Koha! :D (thanks to koha-testing-docker)
And the test plan worked :)
So here is a second signoff while I reach out for help to review the UNIMARC stuff.
Comment 21 Victor Grousset/tuxayo 2021-03-28 10:46:54 UTC
Frido: Will bug 11175 impact this one?
Comment 22 Fridolin Somers 2021-03-29 12:59:51 UTC
(In reply to Victor Grousset/tuxayo from comment #21)
> Frido: Will bug 11175 impact this one?

I think not.
Bug 11175 is a change in XSLT displays.
This one is Zebra configuration.
Comment 23 Victor Grousset/tuxayo 2021-03-29 16:32:22 UTC
> change in XSLT displays

It actually changes biblio-koha-indexdefs.xml but here it's only authority-koha-indexdefs.xml
Comment 24 Martin Renvoize 2021-04-16 11:54:42 UTC
This has had a good number of eyes on it now.. including a Unimarc check.. I'm going to take Victors SO and a QA with agreement from Jonathan on IRC already having been reached.. 

Questions have been answered, QA scripts are happy and this only affects the Unimarc users.

Passing QA
Comment 25 Jonathan Druart 2021-04-16 13:14:03 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 26 Victor Grousset/tuxayo 2021-04-21 20:33:33 UTC
Thanks all. I got help to understand some of the stuff but far from all of it. Also my call for help in the french mailing list didn't yield results ^^"
Comment 27 Fridolin Somers 2021-04-29 09:20:39 UTC
Pushed to 20.11.x for 20.11.06

A great thank you to Victor ;)
Comment 28 Andrew Fuerste-Henry 2021-05-24 15:13:44 UTC
Missing dependency, not backported to 20.05