In marc_defs / unimarc / biblios /, some fields are missing in record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - Identifier-standard : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - Authors : 716, 72X, 730 - Subjects : 608, 615, 616, 617, 620, 621 (and maybe 66X ?) - Title : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t - Publisher : 071b (may contain the name of a music publisher) - Classification codes : 670, 675, 686 Beside : - 207 and 208 are wrong. - 4XX$d are useless : pubdate should be the date of the document itself, not of related titles. - some fields in record.abs do not exist in Unimarc standard : 414-420 ; 603, 630-636 First of all, I will make a patch for record.abs. If it is OK, I will make other patches for the 2 DOM configuration files. (And after that, I will make other patches to improve unimarc indexation) Regards, Mathieu Saby Rennes 2 University
Created attachment 13115 [details] [review] [PATCH] Bug 8984: index new fields in UNIMARC record.abs This patch - add new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - add 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - add new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - correct 207 and 208 - suppress 308a and 328a in Note (useless as complete fields are indexed in same index) - add subfields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t - supress subfields from Pubdate index : 4XX$d (pubdate should be the date of the document itself, not of related titles. Besides, 4XX$d is not used in France) - add (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - add some classifications index : 670, 675, 686 - suppress some non unimarc fields (414-420 ; 603, 630-636) - add some comments (to make easier further modifications) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... If it is correct, I'll make an other patch on the same bug for the 2 DOM indexing configuration files. Mathieu Saby
If some libraries are currently using non-unimarc fields (414-420 ; 603, 630-636) for some reason, I could edit the patch to keep these fields in record.abs, and just add a comment before each field to let people know it is not plain UNIMARC... Mathieu Saby
(In reply to comment #2) > If some libraries are currently using non-unimarc fields (414-420 ; 603, > 630-636) for some reason, I could edit the patch to keep these fields in > record.abs, and just add a comment before each field to let people know it > is not plain UNIMARC... I would prefer this option, especially so close to a release. I look forward to seeing your revised patch.
Created attachment 13269 [details] [review] [PATCH] Bug 8984: index new fields in UNIMARC record.abs (revision 1) I reverted the suppression of non-unimarc fields. In this new patch they are just commented. (and I add 560 field that I did not see last week) M. Saby Rennes 2 University
Created attachment 13270 [details] [review] [PATCH] Bug 8984: index new fields in UNIMARC DOM-indexing configuration files With the new record.abs, I created a new biblio-koha-indexdefs.xml with make_zebra_dom_cfg_from_record_abs, then a new biblio-zebra-indexdefs.xsl with xsltproc, and I made a patch. It must be applied with patch 13269 and tested in with Koha using DOM-indexing. M. Saby Rennes 2 University
Created attachment 13303 [details] [review] [PATCH] Bug 8984: make Zebra more UNIMARC compliant This new patch does not suppress the 4XX$d. This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - adding new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note) - correcting 207 and 208 - suppressing 308a and 328a in Note (useless as complete fields are indexed in same index) - adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560 - adding (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... Mathieu Saby Rennes 2 University
I see the path. For Italian users of Unimarc, it is OK. The patch doesn't change anything used by italian unimarc users.
Thank you Zeno. So could you sign it off ? Mathieu Saby
Hi Mathieu, the patch has problem on master. From sandboxes: Something went wrong ! Applying: Bug 8984: make Zebra more UNIMARC compliant Using index info to reconstruct a base tree... Falling back to patching base and 3-way merge... Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl Failed to merge in the changes. Patch failed at 0001 Bug 8984: make Zebra more UNIMARC compliant When you have resolved this problem run git am --resolved. If you would prefer to skip this patch, instead run git am --skip. To restore the original branch and stop patching run git am --abort. Bug 8984 - Zebra does not index some fields in UNIMARC biblio records I think you need to rebase it
(In reply to comment #9) > Hi Mathieu, > > the patch has problem on master. > From sandboxes: > > Something went wrong ! Hi Zeno & Mathieu, changes related to zebra configuration are *not* managed properly by the sandboxes. It must be done manually. Sorry about that. [ patches about updatedatabase can't too, but this problem will be fixed with the new updatedb mechanism (once it gets in...) ]
Created attachment 13899 [details] [review] [PATCH] Bug 8984: make Zebra more UNIMARC compliant Patch rebased This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - adding new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note) - correcting 207 and 208 - suppressing 308a and 328a in Note (useless as complete fields are indexed in same index) - adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560 - adding (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636, 646) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... - same test with DOM indexing activated
(In fact I am probably wrong saying you need to copy the file in etc/zebradb/biblios/etc In my install, the file was in this directory in addition to etc/zebradb/marc_defs/unimarc/biblios) M. Saby
Created attachment 13965 [details] [review] the entire patch sign-offed It apply correctly to current master
QA comments: I made the following tests : * without patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing OK (can index & search) * with patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing OK (can index & search) * without patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with dom unimarc ==> indexing is KO, I get the following error message: 15:28:48-03/01 zebraidx(25369) [warn] Unknown register type: * after patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing is KO, I get the following error message: 15:12:31-03/01 zebraidx(22028) [warn] Unknown register type: It seems that the DOM indexing in UNIMARC has a problem, not related to this patch. I'll open a bug for this problem, and mark passed QA this patch.
Created attachment 14395 [details] [review] Bug 8984: make Zebra more UNIMARC compliant This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl : - adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z - adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher) - adding new (sub)fields to Author and Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note) - correcting 207 and 208 - suppressing 308a and 328a in Note (useless as complete fields are indexed in same index) - adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560 - adding (sub)fields to Subject and Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621 - adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636, 646) To test : - take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc - try to find the record => not possible - apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra - try to find the record => should be ok - check nothing else is broken... - same test with DOM indexing activated http://bugs.koha-community.org/show_bug.cgi?id=8984 Signed-off-by: Zeno Tajoli <tajoli@cilea.it> Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
(In reply to comment #14) > It seems that the DOM indexing in UNIMARC has a problem, not related to this > patch. I'll open a bug for this problem, and mark passed QA this patch. http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9342
This patch has been pushed to master.
Pushed to 3.10.x and 3.8.x