Bug 28337

Summary: Add System-control-number index for authorities to MARC21 indexes
Product: Koha Reporter: Nick Clemens <nick>
Component: Searching - ZebraAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact:
Severity: enhancement    
Priority: P5 - low CC: dcook, fridolin.somers, kyle
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18540
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
21.11.00
Attachments: Bug 28337: Add System-control-number index to Zebra MARC21 files
Bug 28337: Add System-control-number index to Zebra MARC21 files
Bug 28337: Add System-control-number index to Zebra MARC21 files
Bug 28337: Use existing definition in ccl.properties and bib1.att
Bug 28337: Use existing definition in ccl.properties and bib1.att
Bug 28337: Add System-control-number index to Zebra MARC21 files
Bug 28337: Use existing definition in ccl.properties and bib1.att

Description Nick Clemens 2021-05-12 17:52:02 UTC
https://www.loc.gov/marc/authority/ad035.html

The 035 field in authorities can contain a system control number, this is useful for authority cleanup work as it can contain a non-Koha id for the record

When importing, however, we cannot use this field for matching as it is not indexed
Comment 1 Nick Clemens 2021-05-12 17:58:14 UTC
Created attachment 120894 [details] [review]
Bug 28337: Add System-control-number index to Zebra MARC21 files

To test:
1 - Apply patch
2 - Copy zebra files to destination:
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl
3 - Reindex authorities
4 - Edit an authority and add 035$aExpialodocious
5 - Export the authority
6 - Create a new record matchign rule:
    Matching rule code: SCN
    Description: System control number
    Match threshhold: 1000
    Record type: Authority record
    Search-index: System-control-number
    Score: 1000
    Tag: 035
    Subfields: a
7 - Stage the record and use the new matchign rule
8 - Match found!
Comment 2 Andrew Fuerste-Henry 2021-05-13 18:39:08 UTC
Created attachment 120946 [details] [review]
Bug 28337: Add System-control-number index to Zebra MARC21 files

To test:
1 - Apply patch
2 - Copy zebra files to destination:
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl
3 - Reindex authorities
4 - Edit an authority and add 035$aExpialodocious
5 - Export the authority
6 - Create a new record matchign rule:
    Matching rule code: SCN
    Description: System control number
    Match threshhold: 1000
    Record type: Authority record
    Search-index: System-control-number
    Score: 1000
    Tag: 035
    Subfields: a
7 - Stage the record and use the new matchign rule
8 - Match found!

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 3 Katrin Fischer 2021-07-24 23:34:16 UTC
Hi :)

Some small things:

1) Patch includes an unrelated change to koha-tmpl/intranet-tmpl/prog/en/includes/tools-item-action.inc.

2) Index name: We already have an index on 035 for bibliographic records. My suggestion is to have the index names match, to make them a little easier to work with. It's Other-control-number or ocn.
Comment 4 Nick Clemens 2021-07-26 10:35:43 UTC
Created attachment 123184 [details] [review]
Bug 28337: Add System-control-number index to Zebra MARC21 files

To test:
1 - Apply patch
2 - Copy zebra files to destination:
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl
3 - Reindex authorities
4 - Edit an authority and add 035$aExpialodocious
5 - Export the authority
6 - Create a new record matchign rule:
    Matching rule code: OCN
    Description: Other control number
    Match threshhold: 1000
    Record type: Authority record
    Search-index: Other-control-number
    Score: 1000
    Tag: 035
    Subfields: a
7 - Stage the record and use the new matchign rule
8 - Match found!

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>
Comment 5 Katrin Fischer 2021-08-29 12:30:19 UTC
Hi Nick,

sorry, I might have confused myself a bit here:

In ccl.properties we now have doubled up lines Other-control-number

It looks like there is only one file for both bibliographic and authority indexes? I am not sure about the consequences of this.

We now have:

From before (bibliogrpahic):
#Other-control-number 1211  Other control number            MARC21 035
Other-control-number 1=1211

New (auhority):
Other-control-number 1=Other-control-number

I wonder if these line will be in conflict - we have different use attributes for bibliographic and authority. But only the attribute 1211 is even mentioned in the file, 9052 is not.

Can index names not overlap or should we start using the same attribute for both?

Can you help unravel?
Comment 6 Nick Clemens 2021-08-30 12:34:00 UTC
Created attachment 124240 [details] [review]
Bug 28337: Use existing definition in ccl.properties and bib1.att

As the field is already dfined, we don't need to add anything here.
Bib1.att can use the existing number as well

To test, enable zebra debugging in koha-conf, adding 'request' to the list:
<zebra_loglevels>none,fatal,warn,request</zebra_loglevels>

Restart all the things

Repeat matching (redo matching with no rule, then with OCN rule)
Tail the zebra-output.log and note 1=Ohter-control-number is searched and match is found
Perform a search in the staff client for: other-control-number:expialodocious
Note in logs that 1=1211 is searched

Previous test plan did not mention copying ccl.properties and bib1.att to the package install,
so highlighted that things work without these changes
Comment 7 Katrin Fischer 2021-10-10 15:59:42 UTC
Created attachment 126024 [details] [review]
Bug 28337: Use existing definition in ccl.properties and bib1.att

As the field is already dfined, we don't need to add anything here.
Bib1.att can use the existing number as well

To test, enable zebra debugging in koha-conf, adding 'request' to the list:
<zebra_loglevels>none,fatal,warn,request</zebra_loglevels>

Restart all the things

Repeat matching (redo matching with no rule, then with OCN rule)
Tail the zebra-output.log and note 1=Ohter-control-number is searched and match is found
Perform a search in the staff client for: other-control-number:expialodocious
Note in logs that 1=1211 is searched

Previous test plan did not mention copying ccl.properties and bib1.att to the package install,
so highlighted that things work without these changes

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 8 Katrin Fischer 2021-10-10 16:00:28 UTC
Created attachment 126025 [details] [review]
Bug 28337: Add System-control-number index to Zebra MARC21 files

To test:
1 - Apply patch
2 - Copy zebra files to destination:
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml /etc/koha/zebradb/marc_defs/marc21/authorities/authority-koha-indexdefs.xml
    cp /kohadevbox/koha/etc/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl /etc/koha/zebradb/marc_defs/marc21/authorities/authority-zebra-indexdefs.xsl
3 - Reindex authorities
4 - Edit an authority and add 035$aExpialodocious
5 - Export the authority
6 - Create a new record matchign rule:
    Matching rule code: OCN
    Description: Other control number
    Match threshhold: 1000
    Record type: Authority record
    Search-index: Other-control-number
    Score: 1000
    Tag: 035
    Subfields: a
7 - Stage the record and use the new matchign rule
8 - Match found!

Signed-off-by: Andrew Fuerste-Henry <andrew@bywatersolutions.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 9 Katrin Fischer 2021-10-10 16:00:32 UTC
Created attachment 126026 [details] [review]
Bug 28337: Use existing definition in ccl.properties and bib1.att

As the field is already dfined, we don't need to add anything here.
Bib1.att can use the existing number as well

To test, enable zebra debugging in koha-conf, adding 'request' to the list:
<zebra_loglevels>none,fatal,warn,request</zebra_loglevels>

Restart all the things

Repeat matching (redo matching with no rule, then with OCN rule)
Tail the zebra-output.log and note 1=Ohter-control-number is searched and match is found
Perform a search in the staff client for: other-control-number:expialodocious
Note in logs that 1=1211 is searched

Previous test plan did not mention copying ccl.properties and bib1.att to the package install,
so highlighted that things work without these changes

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 10 Jonathan Druart 2021-10-15 09:29:12 UTC
Pushed to master for 21.11, thanks to everybody involved!
Comment 11 Katrin Fischer 2021-10-17 20:57:53 UTC
*** Bug 12736 has been marked as a duplicate of this bug. ***