Bug 17511 - Change authority linking subfield from $9 to $0
Summary: Change authority linking subfield from $9 to $0
Status: Patch doesn't apply
Alias: None
Product: Koha
Classification: Unclassified
Component: Cataloging (show other bugs)
Version: Main
Hardware: All All
: P5 - low enhancement (vote)
Assignee: Johanna Räisä
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2016-10-27 12:28 UTC by Stefan Berndtsson
Modified: 2023-12-07 20:26 UTC (History)
11 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Medium patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 17511 - Authority linking subfield (107.61 KB, patch)
2019-04-16 10:56 UTC, Johanna Räisä
Details | Diff | Splinter Review
Bug 17511 - Authority linking subfield (76.10 KB, patch)
2020-12-16 11:45 UTC, Johanna Räisä
Details | Diff | Splinter Review
Bug 17511 - Authority linking subfield (76.10 KB, patch)
2020-12-17 08:17 UTC, Johanna Räisä
Details | Diff | Splinter Review
Bug 17511 - Authority linking subfield (648.20 KB, patch)
2021-01-07 12:41 UTC, Johanna Räisä
Details | Diff | Splinter Review
Bug 17511 - Authority linking subfield (76.12 KB, patch)
2021-01-11 07:36 UTC, Johanna Räisä
Details | Diff | Splinter Review
Bug 17511: (DONOTPUSH) Indexing sample files (100.07 KB, patch)
2021-01-11 07:37 UTC, Johanna Räisä
Details | Diff | Splinter Review
Bug 17511: (DONOTPUSH) Indexing sample files (100.23 KB, patch)
2021-02-05 14:49 UTC, Frank Hansen
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Stefan Berndtsson 2016-10-27 12:28:10 UTC
Currently Koha links authority records and bibliographic records using subfield 9. What is the reason behind this choice? According to the MARC21 specification, subfield 0 is "Authority record control number or standard number", which seems more logical.

Usually 9 is a locally defined field, and Libris, the National Union Catalogue in Sweden, uses subfield 9 for various other things already.

What would it take to make the linking subfield into a configurable option?
Comment 1 Zeno Tajoli 2016-10-27 13:14:50 UTC
The selection of $9 was done in the past to have a common subfield with UNIMARC.
In Unimarc the equivalent of $0 is $3.

The main problem to develop a similar request is that this selection is hard-wired into Zebra config files.

So to increase the possibility to develop it, help on Elastichsearch bugs.
Comment 2 Stefan Berndtsson 2016-10-27 13:57:32 UTC
Ok. Thank you.

Just to make it clear to me. Sorry if I'm just repeating the obvious here. The main reason it would be hard to make it into a Koha Preference is that such a setting won't be usable in the Zebra files, but it could be used for ElasticSearch, at least in a possible future?
Comment 3 Zeno Tajoli 2016-10-27 14:14:37 UTC
No problem. It is not so obvious.
The problem is the Zebra config is based on files, very difficult to chage with system preferences. Elastichsearch conf is written into DB, so is much easier to change.
Comment 4 Katrin Fischer 2016-10-27 16:03:28 UTC
I think another reason might be that Koha was using $9 when there was no $0 subfield in MARC21 yet.
Comment 5 Katrin Fischer 2016-10-27 16:05:35 UTC
Hm, I think I confused this with the $w subfields, $0 is indeed quite old:
$0 - Record control number [NEW, 1997]
Comment 6 Stefan Berndtsson 2016-10-28 07:23:52 UTC
We're in the process of testing and managing migration to Koha, and since our current database already contains multiple uses of $9, we may have to implement code changes to move Koha to $0 and change the zebra files as well then. We do intend on using ElasticSearch, but haven't got that bit setup yet.

If we do such a code change, and make the Koha side of the change into a system preference, would that be something that should currently be placed in the Labs section with a big warning sign on it?
Comment 7 Zeno Tajoli 2016-10-28 13:11:16 UTC
If you develop a similar enhancement, start from Elasticsearch setup.
I don't think it is possible to manage Zebra conf files with a system preference.

Where to work:
-- Zebra/Elastichsearch config (indexing)
-- XSLT files
-- cataloguing/addbiblio.pl
-- C4/Biblio.pm
-- C4/AuthoritiesMarc.pm

And probably also in others files.
Comment 8 Magnus Enger 2016-11-09 16:06:13 UTC
Just as a side note, NORMARC also has some uses of $9 that are in conflict with how Koha links to authority records.
Comment 9 Johanna Räisä 2019-03-20 12:06:45 UTC
We are having issues with this in Finland. My opinion is that the authority linking should be changed to $0 permanently. I have some code for it and also script to change the old subfields.
Comment 10 Johanna Räisä 2019-04-16 10:56:15 UTC
Created attachment 88049 [details] [review]
Bug 17511 - Authority linking subfield

Bug 17511: Authority linking subfield

Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0.
This commit will change Koha to use subfield of selected marc flavour. For UNIMARC the subfield is defined to use $9 and for others $0.

To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex.

TEST PLAN:
1) Add new authority linking to a record.
2) Verify that the authority records ID is on $9 subfield.
3) Check index that authority has one record chosen.
4) Apply the patch.
5) Replace zebradb/marc_defs with new files or/and reset ES configurations.
6) Add new authority linking to a record.
7) Verify that the authority records ID is now on the subfield $0.
8) Check index again that authority has record chosen.

Sponsored-by: Koha-Suomi Oy
Comment 11 Katrin Fischer 2019-04-16 11:03:19 UTC
Hi Johanna,

I am not sure the linking script works reliably enough to force this change on libraries that have manually linked all their records

There are also some issues to solve with $0. $0 is repeatable and should contain a prefix with the MARCOrgCode. If we are going to switch because it's standard, I think we need to debate how to make this work.

In our data for example there are 1 to many $0 in one record. Right now for example:

- national authority database number (GND)
- SWB union catalog authority number
- GBV union catalog authority number
- K10plus union catalog authority number (merged catalog of SWB + GBV)

So it might not be as simple as just moving from 9 to 0.
Comment 12 Johanna Räisä 2019-04-16 11:38:06 UTC
The prefix is mainly the indicator from which system the authority has come. Now the script only finds $9 subfield and changes it to $0 and doesn't do anything else so if there are more $0 subfields those are still staying.

The main problem is that now this doesn't support MARC21 format and the $9 subfield is needed for other things.
Comment 13 Katrin Fischer 2019-04-16 14:38:26 UTC
But if it just changes $9 to $0, how do you solve the linking in XSLT?
Comment 14 Katrin Fischer 2019-04-16 14:45:30 UTC
I realize you haven't made any changes to the XSLT files in your patch. They use $9 to create links and woudl need to be adapted. With multiple $0 there will be problems:

- We need to figure out which is the one to use for linking.
- We need to make sure that the indexing is exact enough. Currently () are ignored, so (XXXX)1235 woudl be found when looking for 1235. If you have multiple $0 that could cause issues. Right now, the linking is exact.

I am not saying we shouldn't discuss this, but I think it needs more work.

I see that some authority fields are hardcoded - a lot of libraries seem to put authority control on other fields as well as Koha allows this in configuration.
Comment 15 Johanna Räisä 2019-04-17 05:31:47 UTC
Ahh, missed that. Thank you, will fix that. :)

I even tested with multiple $0 subfields in place. It seemed to work fine. Although there could be some limiting like integer would be the one Koha uses and if there are characters then we assume it is from other source.
Comment 16 Johanna Räisä 2020-12-16 11:45:30 UTC
Created attachment 114433 [details] [review]
Bug 17511 - Authority linking subfield

Bug 17511: Authority linking subfield

Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0.
This commit will add a system preference AuthoritySubfieldLink which is used to define the linking subfield.
The default subfield is set to $9.

To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex.

TEST PLAN:
1) Add new authority linking to a record.
2) Verify that the authority records ID is on $9 subfield.
3) Check index that authority has one record chosen.
4) Apply the patch.
5) Check AuthoritySubfieldLink to be 9.
6) Repeat steps 1, 2 and 3.
7) Change the preference value
8) Replace zebradb/marc_defs with new files or/and reset ES configurations.
9) Add new authority linking to a record.
10) Verify that the authority records ID is now on the chosen subfield.
11) Check index again that authority has record chosen.

Sponsored-by: Koha-Suomi Oy
Comment 17 Johanna Räisä 2020-12-16 11:55:08 UTC
I modified the old patch slightly, now the auhtority linking subfield can be defined in system preferences. The default value is 9 so nothing changes if you don't want to change it to be something else.
Comment 18 David Cook 2020-12-17 06:18:03 UTC
I was thinking about this idea just the other day...

If you look at https://www.loc.gov/marc/bibliographic/ecbdcntf.html, you'll see that $0 can also be a URI. That could be pretty cool. 

It seems too that the $0 is repeatable so that you could have different identifiers... I guess so you could reference VIAF and something else?
Comment 19 David Cook 2020-12-17 06:18:33 UTC
(In reply to Katrin Fischer from comment #4)
> I think another reason might be that Koha was using $9 when there was no $0
> subfield in MARC21 yet.

Bingo.

According to https://www.loc.gov/marc/bibliographic/ecbdcntf.html, the $0 was added (for this purpose) in 2007.
Comment 20 David Cook 2020-12-17 06:23:12 UTC
(In reply to David Cook from comment #18)
> I was thinking about this idea just the other day...
> 
> If you look at https://www.loc.gov/marc/bibliographic/ecbdcntf.html, you'll
> see that $0 can also be a URI. That could be pretty cool. 
> 
> It seems too that the $0 is repeatable so that you could have different
> identifiers... I guess so you could reference VIAF and something else?

We could index a URI, so that Koha's bib to auth link search would still work...

We could actually build in a RDF dereferencing system eventually.
Comment 21 Johanna Räisä 2020-12-17 06:54:10 UTC
(In reply to David Cook from comment #20)
> (In reply to David Cook from comment #18)
> > I was thinking about this idea just the other day...
> > 
> > If you look at https://www.loc.gov/marc/bibliographic/ecbdcntf.html, you'll
> > see that $0 can also be a URI. That could be pretty cool. 
> > 
> > It seems too that the $0 is repeatable so that you could have different
> > identifiers... I guess so you could reference VIAF and something else?
> 
> We could index a URI, so that Koha's bib to auth link search would still
> work...
> 
> We could actually build in a RDF dereferencing system eventually.

Do you mean putting Koha's internal link to authority linking subfield?

We are using national metadata repository and mainly the URI in $0 subfield has a link to that. It might be a good idea that Koha would use URI in authority linking subfield also.

We are facing problem with this $9 subfield because it is used in our national metadata repository for other things and it should not change if we are pulling data to Koha. This is why it would be nice to choose which subfield is used for the authority linking.
Comment 22 Johanna Räisä 2020-12-17 08:17:23 UTC
Created attachment 114474 [details] [review]
Bug 17511 - Authority linking subfield

The patch had wrong author information, replaced with my own.
Comment 23 David Cook 2020-12-18 01:20:09 UTC
(In reply to Johanna Räisä from comment #21)
> Do you mean putting Koha's internal link to authority linking subfield?
> 

I'm not 100% sure what you mean, but I think so, yes.

> We are using national metadata repository and mainly the URI in $0 subfield
> has a link to that. It might be a good idea that Koha would use URI in
> authority linking subfield also.
> 

Yeah that sounds good to me. So you wouldn't be storing authority records in Koha? 

> We are facing problem with this $9 subfield because it is used in our
> national metadata repository for other things and it should not change if we
> are pulling data to Koha. This is why it would be nice to choose which
> subfield is used for the authority linking.

Hmm, how are you pulling the authority data to Koha?
Comment 24 Johanna Räisä 2020-12-21 09:04:48 UTC
(In reply to David Cook from comment #23)
> (In reply to Johanna Räisä from comment #21)
> > Do you mean putting Koha's internal link to authority linking subfield?
> > 
> 
> I'm not 100% sure what you mean, but I think so, yes.

I meant that the authid would be changed to URI and saved that instead of integer value.
> 
> > We are using national metadata repository and mainly the URI in $0 subfield
> > has a link to that. It might be a good idea that Koha would use URI in
> > authority linking subfield also.
> > 
> 
> Yeah that sounds good to me. So you wouldn't be storing authority records in
> Koha? 

No we won't. 

> 
> > We are facing problem with this $9 subfield because it is used in our
> > national metadata repository for other things and it should not change if we
> > are pulling data to Koha. This is why it would be nice to choose which
> > subfield is used for the authority linking.
> 
> Hmm, how are you pulling the authority data to Koha?

We are making a plugin which fetches the authority via SRU or REST API while writing it to the field. We have already a similar approach to get vocabularies https://github.com/KohaSuomi/Koha/commit/9dec82f3c28b41946f8b301dd9200cc36e92cd35
Comment 25 David Cook 2020-12-22 01:40:58 UTC
(In reply to Johanna Räisä from comment #24)
> (In reply to David Cook from comment #23)
> > (In reply to Johanna Räisä from comment #21)
> > > Do you mean putting Koha's internal link to authority linking subfield?
> > > 
> > 
> > I'm not 100% sure what you mean, but I think so, yes.
> 
> I meant that the authid would be changed to URI and saved that instead of
> integer value.
> 

Sounds good to me. 

The only thing to keep in mind is that the $9 is currently indexed into the Koha-Auth-Number:w index which does mean that it's normalized and tokenized. This could cause problems if there is an unencoded space in the URI for instance. 

It might be wise for us to start using a "p" register instead of a "w" register, and to start doing "exact" searches when querying this index... 

At least if you're using Zebra. I don't know about Elasticsearch...

> > Yeah that sounds good to me. So you wouldn't be storing authority records in
> > Koha? 
> 
> No we won't. 
> 

Interesting! Very interesting!

> We are making a plugin which fetches the authority via SRU or REST API while
> writing it to the field. We have already a similar approach to get
> vocabularies
> https://github.com/KohaSuomi/Koha/commit/
> 9dec82f3c28b41946f8b301dd9200cc36e92cd35

Sounds interesting. I'd like to hear more as you progress.

When you say that the plugin fetches the authority, do you mean that it fetches it, extracts the relevant data, inserts it into the Koha bibliographic record, and then discards the authority record?

I think that's such a great idea.
Comment 26 David Cook 2020-12-22 01:51:17 UTC
(In reply to Johanna Räisä from comment #16)
> TEST PLAN:
> 1) Add new authority linking to a record.
> 2) Verify that the authority records ID is on $9 subfield.
> 3) Check index that authority has one record chosen.
> 4) Apply the patch.
> 5) Check AuthoritySubfieldLink to be 9.
> 6) Repeat steps 1, 2 and 3.
> 7) Change the preference value
> 8) Replace zebradb/marc_defs with new files or/and reset ES configurations.
> 9) Add new authority linking to a record.
> 10) Verify that the authority records ID is now on the chosen subfield.
> 11) Check index again that authority has record chosen.
> 

Could you elaborate on Step 8? 

I think it would be wise to add an additional example patch (marked with something like DONOTPUSH) with your desired configuration for etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl

That'll make it easier for us to test. Otherwise, the tester and QA have to guess at the correct configuration. 

I think that this sounds like a good change though. Wise to keep the $9 for default, and maybe we can announce a deprecation in favour of moving to $0 at some point...
Comment 27 Johanna Räisä 2021-01-07 12:41:31 UTC
Created attachment 114918 [details] [review]
Bug 17511 - Authority linking subfield

Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0.
This commit will add a system preference AuthoritySubfieldLink which is used to define the linking subfield.
The default subfield is set to $9.

To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex.

TEST PLAN:
1) Add new authority linking to a record.
2) Verify that the authority records ID is on $9 subfield.
3) Check index that authority has one record chosen.
4) Apply the patch.
5) Check AuthoritySubfieldLink to be 9.
6) Repeat steps 1, 2 and 3.
7) Change the preference value to 0
8) Replace zebra defs and elastic mapping file with the commit sample files for testing.
9) Add new authority linking to a record.
10) Verify that the authority records ID is now on the $0 subfield.
11) Check index again that authority has record chosen.

Sponsored-by: Koha-Suomi Oy
Comment 28 Johanna Räisä 2021-01-07 12:46:24 UTC
(In reply to Johanna Räisä from comment #27)
> Created attachment 114918 [details] [review] [review]
> Bug 17511 - Authority linking subfield
> 
> Koha uses $9 subfield for linking authority records. MARC21 format says the
> subfield should be $0.
> This commit will add a system preference AuthoritySubfieldLink which is used
> to define the linking subfield.
> The default subfield is set to $9.
> 
> To change the old linkings run batchAuthorityLinking.pl, modify index
> mappings and reindex.
> 
> TEST PLAN:
> 1) Add new authority linking to a record.
> 2) Verify that the authority records ID is on $9 subfield.
> 3) Check index that authority has one record chosen.
> 4) Apply the patch.
> 5) Check AuthoritySubfieldLink to be 9.
> 6) Repeat steps 1, 2 and 3.
> 7) Change the preference value to 0
> 8) Replace zebra defs and elastic mapping file with the commit sample files
> for testing.
> 9) Add new authority linking to a record.
> 10) Verify that the authority records ID is now on the $0 subfield.
> 11) Check index again that authority has record chosen.
> 
> Sponsored-by: Koha-Suomi Oy

I added only indexing file samples to this patch. Have to think how the URI would work with indexing.
Comment 29 David Cook 2021-01-07 22:28:20 UTC
(In reply to Johanna Räisä from comment #28)
> I added only indexing file samples to this patch. 

My apologies, Johanna. I meant that the indexing file samples should be in a separate commit/patch and that the commit message should include DONOTPUSH at the start of the commit message. The patch you've uploaded which includes both the code changes and the samples is problematic for the release manager and release maintainers. They can't just ignore the patch with the samples. 

Could you split up the patch and re-upload?
Comment 30 David Cook 2021-01-07 22:46:44 UTC
(In reply to Johanna Räisä from comment #28)
> Have to think how the URI would work with indexing.

I have been thinking about the same thing. 

At a glance, with Zebra we're using Koha-Auth-Number with a "w" register, which is fine for integers, but that probably won't work very well with URIs, since it's for storing "words". It's normalized and tokenized. That will lead to problematic indexing.

We could use a "p" register which is normalized and not tokenized but it's possible for 2 different URIs to be normalized into the same data, so that's not ideal either.

Perhaps we should use a "0" register which is not normalized and not tokenized. To use that we'd need to use the 4=3 structure attribute and I don't think we'd be using it as the Bib-1 spec says it should be used (https://www.loc.gov/z3950/agency/bib1.html), but I think it would work with Zebra. I'd have to try it out though.

Technically, the is a special "u" register for URLs, but we also need to optionally store integers, so that won't work either. 

I don't use Elasticsearch, so I can't speak to it.
Comment 31 Johanna Räisä 2021-01-11 06:23:35 UTC
(In reply to David Cook from comment #29)
> (In reply to Johanna Räisä from comment #28)
> > I added only indexing file samples to this patch. 
> 
> My apologies, Johanna. I meant that the indexing file samples should be in a
> separate commit/patch and that the commit message should include DONOTPUSH
> at the start of the commit message. The patch you've uploaded which includes
> both the code changes and the samples is problematic for the release manager
> and release maintainers. They can't just ignore the patch with the samples. 
> 
> Could you split up the patch and re-upload?

Oh, you ment that. Sure, I will split them.
Comment 32 Johanna Räisä 2021-01-11 07:36:38 UTC
Created attachment 115002 [details] [review]
Bug 17511 - Authority linking subfield

Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0.
This commit will add a system preference AuthoritySubfieldLink which is used to define the linking subfield.
The default subfield is set to $9.

To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex.

TEST PLAN:
1) Add new authority linking to a record.
2) Verify that the authority records ID is on $9 subfield.
3) Check index that authority has one record chosen.
4) Apply the patch.
5) Check AuthoritySubfieldLink to be 9.
6) Repeat steps 1, 2 and 3.
7) Change the preference value to 0
8) Replace zebra defs and elastic mapping file with the sample files from second patch for testing.
9) Add new authority linking to a record.
10) Verify that the authority records ID is now on the $0 subfield.
11) Check index again that authority has record chosen.

Sponsored-by: Koha-Suomi Oy
Comment 33 Johanna Räisä 2021-01-11 07:37:53 UTC
Created attachment 115003 [details] [review]
Bug 17511: (DONOTPUSH) Indexing sample files

These are the indexing sample files for testing AuthoritySubfieldLink systempreference.
Comment 34 Johanna Räisä 2021-01-11 08:14:12 UTC
(In reply to David Cook from comment #30)
> (In reply to Johanna Räisä from comment #28)
> > Have to think how the URI would work with indexing.
> 
> I have been thinking about the same thing. 
> 
> At a glance, with Zebra we're using Koha-Auth-Number with a "w" register,
> which is fine for integers, but that probably won't work very well with
> URIs, since it's for storing "words". It's normalized and tokenized. That
> will lead to problematic indexing.
> 
> We could use a "p" register which is normalized and not tokenized but it's
> possible for 2 different URIs to be normalized into the same data, so that's
> not ideal either.
> 
> Perhaps we should use a "0" register which is not normalized and not
> tokenized. To use that we'd need to use the 4=3 structure attribute and I
> don't think we'd be using it as the Bib-1 spec says it should be used
> (https://www.loc.gov/z3950/agency/bib1.html), but I think it would work with
> Zebra. I'd have to try it out though.

This sounds good to me, although I have little experience configuring Zebra. 

> 
> Technically, the is a special "u" register for URLs, but we also need to
> optionally store integers, so that won't work either. 
> 
> I don't use Elasticsearch, so I can't speak to it.

I can look Elasticsearch. I have work with it more than I have worked with Zebra.

Maybe we should make another bug for this? At least I think this is another enhancement.
Comment 35 David Cook 2021-01-14 04:15:57 UTC
I'm intrigued by this one but super busy at the moment, unfortunately.
Comment 36 Frank Hansen 2021-02-05 14:49:31 UTC
Created attachment 116405 [details] [review]
Bug 17511: (DONOTPUSH) Indexing sample files

These are the indexing sample files for testing AuthoritySubfieldLink systempreference.

Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se>

This patch works fine in elasticsearch at least. Great job!
Comment 37 Phil Ringnalda 2021-02-26 02:43:14 UTC
It would be a good idea to remove the last mentions of $0 from the patch, and also from the bug summary, since this isn't at all a $0 implementation. $0 is a repeatable subfield, with specified contents of "(MARC org code from 003 in an authority record)control number of an authority record", which isn't what this is doing.
Comment 38 David Cook 2021-02-26 03:39:40 UTC
(In reply to Phil Ringnalda from comment #37)
> It would be a good idea to remove the last mentions of $0 from the patch,
> and also from the bug summary, since this isn't at all a $0 implementation.
> $0 is a repeatable subfield, with specified contents of "(MARC org code from
> 003 in an authority record)control number of an authority record", which
> isn't what this is doing.

I haven't reviewed these patches in a while, so I can't speak to the first part of that.

However, note that $0 is more complex than "(MARC org code from 003 in an authority record)control number of an authority record". See "$0 - Authority record control number or standard number" at the following URL for more details: https://www.loc.gov/marc/bibliographic/ecbdcntf.html
Comment 39 Esther Melander 2023-12-07 20:26:37 UTC
To add a wrinkle to this discussion. OCLC is using $0 in authority records, perhaps as intended with linking information. An example authority:

Conduct of life. $2 fast $0 (OCoLC)fst00874563

In this case, a library wants to retain this subfield as a reference point. It would cause a conflict if a library is using $0 for internal linking and brought in a record which contains reference data in this field for a different system.