Bug 8984 - Zebra does not index some fields in UNIMARC biblio records
Summary: Zebra does not index some fields in UNIMARC biblio records
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: mathieu saby
QA Contact: Paul Poulain
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2012-10-30 11:23 UTC by mathieu saby
Modified: 2014-05-26 21:05 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Medium patch
Who signed the patch off:
Text to go in the release notes:
Version(s) released in:


Attachments
[PATCH] Bug 8984: index new fields in UNIMARC record.abs (16.46 KB, patch)
2012-10-30 13:15 UTC, mathieu saby
Details | Diff | Splinter Review
[PATCH] Bug 8984: index new fields in UNIMARC record.abs (revision 1) (17.90 KB, patch)
2012-11-06 16:51 UTC, mathieu saby
Details | Diff | Splinter Review
[PATCH] Bug 8984: index new fields in UNIMARC DOM-indexing configuration files (175.05 KB, patch)
2012-11-06 17:00 UTC, mathieu saby
Details | Diff | Splinter Review
[PATCH] Bug 8984: make Zebra more UNIMARC compliant (192.86 KB, patch)
2012-11-08 09:22 UTC, mathieu saby
Details | Diff | Splinter Review
[PATCH] Bug 8984: make Zebra more UNIMARC compliant (129.05 KB, patch)
2012-12-06 00:04 UTC, mathieu saby
Details | Diff | Splinter Review
the entire patch sign-offed (129.04 KB, patch)
2012-12-09 13:13 UTC, Zeno Tajoli
Details | Diff | Splinter Review
Bug 8984: make Zebra more UNIMARC compliant (129.20 KB, patch)
2013-01-03 14:59 UTC, Paul Poulain
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description mathieu saby 2012-10-30 11:23:29 UTC
In marc_defs / unimarc / biblios /, some fields are missing in record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl :

- Identifier-standard : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z
- Authors : 716, 72X, 730
- Subjects : 608, 615, 616, 617, 620, 621 (and maybe 66X ?)
- Title  : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t
- Publisher : 071b (may contain the name of a music publisher)
- Classification codes : 670, 675, 686

Beside :
- 207 and 208 are wrong.
- 4XX$d are useless : pubdate should be the date of the document itself, not of related titles.
- some fields in record.abs do not exist in Unimarc standard : 414-420 ; 603, 630-636

First of all, I will make a patch for record.abs.
If it is OK, I will make other patches for the 2 DOM configuration files.

(And after that, I will make other patches to improve unimarc indexation)

Regards,
Mathieu Saby
Rennes 2 University
Comment 1 mathieu saby 2012-10-30 13:15:30 UTC Comment hidden (obsolete)
Comment 2 mathieu saby 2012-10-30 14:57:20 UTC
If some libraries are currently using non-unimarc fields (414-420 ; 603, 630-636) for some reason, I could edit the patch to keep these fields in record.abs, and just add a comment before each field to let people know it is not plain UNIMARC...

Mathieu Saby
Comment 3 Jared Camins-Esakov 2012-11-01 19:54:13 UTC
(In reply to comment #2)
> If some libraries are currently using non-unimarc fields (414-420 ; 603,
> 630-636) for some reason, I could edit the patch to keep these fields in
> record.abs, and just add a comment before each field to let people know it
> is not plain UNIMARC...

I would prefer this option, especially so close to a release. I look forward to seeing your revised patch.
Comment 4 mathieu saby 2012-11-06 16:51:26 UTC Comment hidden (obsolete)
Comment 5 mathieu saby 2012-11-06 17:00:30 UTC Comment hidden (obsolete)
Comment 6 mathieu saby 2012-11-08 09:22:51 UTC Comment hidden (obsolete)
Comment 7 Zeno Tajoli 2012-12-05 08:34:56 UTC
I see the path.
For Italian users of Unimarc, it is OK.
The patch doesn't change anything used by italian unimarc users.
Comment 8 mathieu saby 2012-12-05 10:25:00 UTC
Thank you Zeno. So could you sign it off ?
Mathieu Saby
Comment 9 Zeno Tajoli 2012-12-05 11:30:43 UTC
Hi Mathieu,

the patch has problem on master.
From sandboxes:

Something went wrong !

Applying: Bug 8984: make Zebra more UNIMARC compliant
Using index info to reconstruct a base tree...
Falling back to patching base and 3-way merge...
Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml
CONFLICT (content): Merge conflict in etc/zebradb/marc_defs/unimarc/biblios/biblio-koha-indexdefs.xml
Auto-merging etc/zebradb/marc_defs/unimarc/biblios/biblio-zebra-indexdefs.xsl
Failed to merge in the changes.
Patch failed at 0001 Bug 8984: make Zebra more UNIMARC compliant
When you have resolved this problem run git am --resolved.
If you would prefer to skip this patch, instead run git am --skip.
To restore the original branch and stop patching run git am --abort.
Bug 8984 - Zebra does not index some fields in UNIMARC biblio records

I think you need to rebase it
Comment 10 Paul Poulain 2012-12-05 13:38:09 UTC
(In reply to comment #9)
> Hi Mathieu,
> 
> the patch has problem on master.
> From sandboxes:
> 
> Something went wrong !
Hi Zeno & Mathieu,

changes related to zebra configuration are *not* managed properly by the sandboxes. It must be done manually.

Sorry about that.

[ patches about updatedatabase can't too, but this problem will be fixed with the new updatedb mechanism (once it gets in...) ]
Comment 11 mathieu saby 2012-12-06 00:04:54 UTC Comment hidden (obsolete)
Comment 12 mathieu saby 2012-12-06 17:04:06 UTC
(In fact I am probably wrong saying you need to copy the file in etc/zebradb/biblios/etc
In my install, the file was in this directory in addition to etc/zebradb/marc_defs/unimarc/biblios)

M. Saby
Comment 13 Zeno Tajoli 2012-12-09 13:13:27 UTC Comment hidden (obsolete)
Comment 14 Paul Poulain 2013-01-03 14:57:23 UTC
QA comments:
 I made the following tests :
  * without patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing OK (can index & search)
  * with patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing OK (can index & search)

  * without patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with dom unimarc ==> indexing is KO, I get the following error message: 15:28:48-03/01 zebraidx(25369) [warn] Unknown register type: 
  * after patch = rm -rf $KOHA_CONF_DIR && perl Makefile.PL && make install with grs1 unimarc ==> indexing is KO, I get the following error message:
15:12:31-03/01 zebraidx(22028) [warn] Unknown register type: 


It seems that the DOM indexing in UNIMARC has a problem, not related to this patch. I'll open a bug for this problem, and mark passed QA this patch.
Comment 15 Paul Poulain 2013-01-03 14:59:21 UTC
Created attachment 14395 [details] [review]
Bug 8984: make Zebra more UNIMARC compliant

This patch makes the following changes to record.abs, biblio-koha-indexdefs.xml and biblio-zebra-indexdefs.xsl :
- adding new (sub)fields to Identifier-standard index : 011f/g ; 012a ; 013a/z ; 014a/z ; 015a/z ; 016a/z ; 017a/z, 040a/z, 071z, 072z, 073z
- adding 1 new subfield to Publisher index : 071b (may contain the name of a music publisher)
- adding new (sub)fields to Author and  Identifier-standard index (for the $9) : 716, 72X, 730 - adding new (sub)fields to Note : 334$a (award note)
- correcting 207 and 208
- suppressing 308a and 328a in Note (useless as complete fields are indexed in same index)
- adding (sub)fields to Title index : 411t, 421-425t, 433-437t, 442-444t, 446-456t, 462-463t, 470-488t, 560
- adding (sub)fields to Subject and  Identifier-standard index (for the $9) : 608, 615, 616, 617, 620, 621
- adding some classifications index : 670, 675, 686 - adding some comments (to make easier further modifications and to identify non unimarc fields : 414-420, 603, 630-636, 646)

To test :
- take a record and fill some of the missing fields (e.g 488t, 608, 720, 012a) with some data as "field488", "field608" etc
- try to find the record => not possible
- apply the patch, copy the new record.abs in etc/zebradb/biblios/etc and rebuild zebra
- try to find the record => should be ok
- check nothing else is broken...
- same test with DOM indexing activated

http://bugs.koha-community.org/show_bug.cgi?id=8984
Signed-off-by: Zeno Tajoli <tajoli@cilea.it>
Signed-off-by: Paul Poulain <paul.poulain@biblibre.com>
Comment 16 Paul Poulain 2013-01-03 15:02:24 UTC
(In reply to comment #14)
> It seems that the DOM indexing in UNIMARC has a problem, not related to this
> patch. I'll open a bug for this problem, and mark passed QA this patch.

http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=9342
Comment 17 Jared Camins-Esakov 2013-01-04 13:40:06 UTC
This patch has been pushed to master.
Comment 18 Chris Cormack 2013-01-12 19:12:44 UTC
Pushed to 3.10.x and 3.8.x