This is a unimarc bug only: When importing biblios from Z39.50, related authorities are wrong matched by the linker. This causes duplicates of authorities. The problem come from a bad marc definition in unimarc/authorities/authority-koha-indexdefs.xml which does not create a index for matching an entire heading.
Created attachment 59675 [details] [review] Bug 18017 - add index_matching_heading to create indexes for matching an entire authority heading
A test plan to help testers?
If I understand the patch, a test plan could be Test plan BiblioAddsAuthorities ON AutoCreateAuthorities ON 1) Import Z3950 record and save it : authorities are created 2) Import the same record again, when saving say it is not a duplicate : Without the patch : new authorities (duplicates) are created for the same values With the patch : new record should be linked with existing authorities Tested on sandbox : BiblioAddsAuthorities ON AutoCreateAuthorities ON 1) Import Z3950 record and save it : authorities are created 2) Import the same record again, when saving say it is not a duplicate : Without the patch : new authorities (duplicates) are created for the same values With the patch : new authorities (duplicates) are created for the same values : KO
Full test plan is: BiblioAddsAuthorities ON AutoCreateAuthorities ON 1) replace /koha_conf_dir/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml with /koha_clone/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml 2) replace /koha_conf_dir/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl with /koha_clone/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl 3) reindex authorities, 4) Import Z3950 record and save it : authorities are created 5) Import the same record again, when saving say it is not a duplicate : Without the patch : new authorities (duplicates) are created for the same values With the patch : new record should be linked with existing authorities
After applying the patch, when reimporting, the authority of the main author is still duplicated. (different id on the "Main Author" link) I even re-indexed authorities after applying the patch. ./src/misc/migration_tools/rebuild_zebra.pl -r -a -v
Retried: 1) Set sysprefs BiblioAddsAuthorities ON AutoCreateAuthorities ON 2) Replace xsl files cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl 3) reindex authorities ./misc/migration_tools/rebuild_zebra.pl -r -a -v 4) Import Z3950 record and save it : authorities are created go to staff:/cgi-bin/koha/cataloguing/addbooks.pl 5) Reimport the same record (when asked, say that it's not a duplicate) 6) The authority should have been duplicated (different url) 7) Apply this patch 8) Replace again the xsl files 9) reindex authorities 10) Reimport the same record 11) The authority should have not been duplicated But for me it was.
Followed testing steps, but authority is duplicated. Added book "Sometimes I Lie" twice, and got two "Feeney, Alice, author" records in authorities.
Hi I can only reproduce https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18017#c7 on MARC21 and it's solved by 21958, but this patch is about UNIMARC and it's working for me, so back to needs Signoff. Regards Didier
As it's an UNIMARC bug and it's getting old, I think it's okay to have a signoff from the same entity, so Didier or someone else at BibLibre, you can sign this off.
I'm on creating more patches so I set to ASSIGNED
Created attachment 114588 [details] [review] Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration Like for MARC21, UNIMARC authorities has subdivisions form, general, chronological and geographic. In C4::Heading::UNIMARC, use subdivisions in _get_search_heading like in C4::Heading::MARC21. Adds subdivisions variables into UNIMARC authorities zebra configuration. Note that unlike MARC21 geographic is subfield $y and chronological is subfield $z. See https://www.ifla.org/publications/unimarc-formats-and-related-documentation
Created attachment 114589 [details] [review] Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration For a good management of autorities linking to biblio records, MARC21 uses index_heading and index_match_heading in authorities zebra configuration. UNIMARC configuration must use the same. This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading in order to be maximum close to MARC21 authorities zebra configuration. See changes made in MARC21 : https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896 It fixes some indexes names : Personal-name-see => Personal-name-see-from Removes useless Term-geographic index, a duplicate of Name-geographic. Sometimes parallel 7xx form whas only on $a, it must contains same subfields has the main heading. Test plan : =========== 1.0) Use a UNIMARC install without patch 1.1) Set sysprefs BiblioAddsAuthorities = ON AutoCreateAuthorities = ON LinkerModule = First Match 1.2) Replace authorities zebra configuration files cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl 1.3) Restart zebra server and indexer services 1.4) Reindex authorities ./misc/migration_tools/rebuild_zebra.pl -r -a -v 1.5) Search in Z3950 a record with complex heading (with subdivisions), for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues" 1.6) Import this record and save it : authorities are created go to staff:/cgi-bin/koha/cataloguing/addbooks.pl 1.7) Reimport the same record (when asked, say that it's not a duplicate) 1.8) The authority should have been duplicated : different url and different $9 value 2.0) Apply this patch 2.1) Replace again the authorities zebra configuration files 2.2) Restart zebra server and indexer services 2.3) Reindex authorities 2.4) Reimport the same record 2.5) The authority should have not been duplicated 3.0) Play with authorities search to check every mode : Search main heading ($a only) Search main heading Search all headings Search entire record
Created attachment 114590 [details] [review] Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration For a good management of autorities linking to biblio records, MARC21 uses index_heading and index_match_heading in authorities zebra configuration. UNIMARC configuration must use the same. This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading in order to be maximum close to MARC21 authorities zebra configuration. See changes made in MARC21 : https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896 It fixes some indexes names : Personal-name-see => Personal-name-see-from Removes useless Term-geographic index, a duplicate of Name-geographic. Sometimes parallel 7xx form whas only on $a, it must contains same subfields has the main heading. Test plan : =========== 1.0) Use a UNIMARC install without patch 1.1) Set sysprefs BiblioAddsAuthorities = ON AutoCreateAuthorities = ON LinkerModule = First Match 1.2) Replace authorities zebra configuration files cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl 1.3) Restart zebra server and indexer services 1.4) Reindex authorities ./misc/migration_tools/rebuild_zebra.pl -r -a -v 1.5) Search in Z3950 a record with complex heading (with subdivisions), for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues" 1.6) Import this record and save it : authorities are created go to staff:/cgi-bin/koha/cataloguing/addbooks.pl 1.7) Reimport the same record (when asked, say that it's not a duplicate) 1.8) The authority should have been duplicated : different url and different $9 value 2.0) Apply this patch 2.1) Replace again the authorities zebra configuration files 2.2) Restart zebra server and indexer services 2.3) Reindex authorities 2.4) Reimport the same record 2.5) The authority should have not been duplicated 3.0) Play with authorities search to check every mode : Search main heading ($a only) Search main heading Search all headings Search entire record
See also https://github.com/indexdata/idzebra/issues/24
Created attachment 117182 [details] [review] Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration Like for MARC21, UNIMARC authorities has subdivisions form, general, chronological and geographic. In C4::Heading::UNIMARC, use subdivisions in _get_search_heading like in C4::Heading::MARC21. Adds subdivisions variables into UNIMARC authorities zebra configuration. Note that unlike MARC21 geographic is subfield $y and chronological is subfield $z. See https://www.ifla.org/publications/unimarc-formats-and-related-documentation Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 117183 [details] [review] Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration For a good management of autorities linking to biblio records, MARC21 uses index_heading and index_match_heading in authorities zebra configuration. UNIMARC configuration must use the same. This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading in order to be maximum close to MARC21 authorities zebra configuration. See changes made in MARC21 : https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896 It fixes some indexes names : Personal-name-see => Personal-name-see-from Removes useless Term-geographic index, a duplicate of Name-geographic. Sometimes parallel 7xx form whas only on $a, it must contains same subfields has the main heading. Test plan : =========== 1.0) Use a UNIMARC install without patch 1.1) Set sysprefs BiblioAddsAuthorities = ON AutoCreateAuthorities = ON LinkerModule = First Match 1.2) Replace authorities zebra configuration files cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl 1.3) Restart zebra server and indexer services 1.4) Reindex authorities ./misc/migration_tools/rebuild_zebra.pl -r -a -v 1.5) Search in Z3950 a record with complex heading (with subdivisions), for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues" 1.6) Import this record and save it : authorities are created go to staff:/cgi-bin/koha/cataloguing/addbooks.pl 1.7) Reimport the same record (when asked, say that it's not a duplicate) 1.8) The authority should have been duplicated : different url and different $9 value 2.0) Apply this patch 2.1) Replace again the authorities zebra configuration files 2.2) Restart zebra server and indexer services 2.3) Reindex authorities 2.4) Reimport the same record 2.5) The authority should have not been duplicated 3.0) Play with authorities search to check every mode : Search main heading ($a only) Search main heading Search all headings Search entire record Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
OK, I'm pretty familiar with the code around this in MARC21 and it looks like a reasonably good representation for UNIMARC here.. I did some testing to the extent I understand UNIMARC and it all works well.. Signing off.. I'll request Victor to take a look from a QA standpoint, hopefully he'll spot any UNIMARC issues I may not and can get it over the finish line.
Created attachment 117350 [details] [review] Bug 18017: Add subdivisions to UNIMARC authorities zebra configuration Like for MARC21, UNIMARC authorities has subdivisions form, general, chronological and geographic. In C4::Heading::UNIMARC, use subdivisions in _get_search_heading like in C4::Heading::MARC21. Adds subdivisions variables into UNIMARC authorities zebra configuration. Note that unlike MARC21 geographic is subfield $y and chronological is subfield $z. See https://www.ifla.org/publications/unimarc-formats-and-related-documentation Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Created attachment 117351 [details] [review] Bug 18017: Use index_heading and index_match_heading in UNIMARC authorities zebra configuration For a good management of autorities linking to biblio records, MARC21 uses index_heading and index_match_heading in authorities zebra configuration. UNIMARC configuration must use the same. This patch adds in UNIMARC authorities zebra configuration index_heading and index_match_heading to earch heading in order to be maximum close to MARC21 authorities zebra configuration. See changes made in MARC21 : https://git.koha-community.org/Koha-community/Koha/commit/32cf2af700dfea15d9c4f99bee97e97c85643896 It fixes some indexes names : Personal-name-see => Personal-name-see-from Removes useless Term-geographic index, a duplicate of Name-geographic. Sometimes parallel 7xx form whas only on $a, it must contains same subfields has the main heading. Test plan : =========== 1.0) Use a UNIMARC install without patch 1.1) Set sysprefs BiblioAddsAuthorities = ON AutoCreateAuthorities = ON LinkerModule = First Match 1.2) Replace authorities zebra configuration files cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-koha-indexdefs.xml cp $KOHA_CLONE/etc/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl $KOHA_CONF_DIR/zebradb/marc_defs/unimarc/authorities/authority-zebra-indexdefs.xsl 1.3) Restart zebra server and indexer services 1.4) Reindex authorities ./misc/migration_tools/rebuild_zebra.pl -r -a -v 1.5) Search in Z3950 a record with complex heading (with subdivisions), for example ISBN 2877620115 "Facteurs culturels et sociaux de la santé en Afrique de l'Oues" 1.6) Import this record and save it : authorities are created go to staff:/cgi-bin/koha/cataloguing/addbooks.pl 1.7) Reimport the same record (when asked, say that it's not a duplicate) 1.8) The authority should have been duplicated : different url and different $9 value 2.0) Apply this patch 2.1) Replace again the authorities zebra configuration files 2.2) Restart zebra server and indexer services 2.3) Reindex authorities 2.4) Reimport the same record 2.5) The authority should have not been duplicated. Compare with both existing records to see which the 3rd has been matched against. 3.0) Play with authorities search to check every mode : Search main heading ($a only) Search main heading Search all headings Search entire record Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Finally I have an UNIMARC Koha! :D (thanks to koha-testing-docker) And the test plan worked :) So here is a second signoff while I reach out for help to review the UNIMARC stuff.
Frido: Will bug 11175 impact this one?
(In reply to Victor Grousset/tuxayo from comment #21) > Frido: Will bug 11175 impact this one? I think not. Bug 11175 is a change in XSLT displays. This one is Zebra configuration.
> change in XSLT displays It actually changes biblio-koha-indexdefs.xml but here it's only authority-koha-indexdefs.xml
This has had a good number of eyes on it now.. including a Unimarc check.. I'm going to take Victors SO and a QA with agreement from Jonathan on IRC already having been reached.. Questions have been answered, QA scripts are happy and this only affects the Unimarc users. Passing QA
Pushed to master for 21.05, thanks to everybody involved!
Thanks all. I got help to understand some of the stuff but far from all of it. Also my call for help in the french mailing list didn't yield results ^^"
Pushed to 20.11.x for 20.11.06 A great thank you to Victor ;)
Missing dependency, not backported to 20.05