Currently Koha links authority records and bibliographic records using subfield 9. What is the reason behind this choice? According to the MARC21 specification, subfield 0 is "Authority record control number or standard number", which seems more logical. Usually 9 is a locally defined field, and Libris, the National Union Catalogue in Sweden, uses subfield 9 for various other things already. What would it take to make the linking subfield into a configurable option?
The selection of $9 was done in the past to have a common subfield with UNIMARC. In Unimarc the equivalent of $0 is $3. The main problem to develop a similar request is that this selection is hard-wired into Zebra config files. So to increase the possibility to develop it, help on Elastichsearch bugs.
Ok. Thank you. Just to make it clear to me. Sorry if I'm just repeating the obvious here. The main reason it would be hard to make it into a Koha Preference is that such a setting won't be usable in the Zebra files, but it could be used for ElasticSearch, at least in a possible future?
No problem. It is not so obvious. The problem is the Zebra config is based on files, very difficult to chage with system preferences. Elastichsearch conf is written into DB, so is much easier to change.
I think another reason might be that Koha was using $9 when there was no $0 subfield in MARC21 yet.
Hm, I think I confused this with the $w subfields, $0 is indeed quite old: $0 - Record control number [NEW, 1997]
We're in the process of testing and managing migration to Koha, and since our current database already contains multiple uses of $9, we may have to implement code changes to move Koha to $0 and change the zebra files as well then. We do intend on using ElasticSearch, but haven't got that bit setup yet. If we do such a code change, and make the Koha side of the change into a system preference, would that be something that should currently be placed in the Labs section with a big warning sign on it?
If you develop a similar enhancement, start from Elasticsearch setup. I don't think it is possible to manage Zebra conf files with a system preference. Where to work: -- Zebra/Elastichsearch config (indexing) -- XSLT files -- cataloguing/addbiblio.pl -- C4/Biblio.pm -- C4/AuthoritiesMarc.pm And probably also in others files.
Just as a side note, NORMARC also has some uses of $9 that are in conflict with how Koha links to authority records.
We are having issues with this in Finland. My opinion is that the authority linking should be changed to $0 permanently. I have some code for it and also script to change the old subfields.
Created attachment 88049 [details] [review] Bug 17511 - Authority linking subfield Bug 17511: Authority linking subfield Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0. This commit will change Koha to use subfield of selected marc flavour. For UNIMARC the subfield is defined to use $9 and for others $0. To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex. TEST PLAN: 1) Add new authority linking to a record. 2) Verify that the authority records ID is on $9 subfield. 3) Check index that authority has one record chosen. 4) Apply the patch. 5) Replace zebradb/marc_defs with new files or/and reset ES configurations. 6) Add new authority linking to a record. 7) Verify that the authority records ID is now on the subfield $0. 8) Check index again that authority has record chosen. Sponsored-by: Koha-Suomi Oy
Hi Johanna, I am not sure the linking script works reliably enough to force this change on libraries that have manually linked all their records There are also some issues to solve with $0. $0 is repeatable and should contain a prefix with the MARCOrgCode. If we are going to switch because it's standard, I think we need to debate how to make this work. In our data for example there are 1 to many $0 in one record. Right now for example: - national authority database number (GND) - SWB union catalog authority number - GBV union catalog authority number - K10plus union catalog authority number (merged catalog of SWB + GBV) So it might not be as simple as just moving from 9 to 0.
The prefix is mainly the indicator from which system the authority has come. Now the script only finds $9 subfield and changes it to $0 and doesn't do anything else so if there are more $0 subfields those are still staying. The main problem is that now this doesn't support MARC21 format and the $9 subfield is needed for other things.
But if it just changes $9 to $0, how do you solve the linking in XSLT?
I realize you haven't made any changes to the XSLT files in your patch. They use $9 to create links and woudl need to be adapted. With multiple $0 there will be problems: - We need to figure out which is the one to use for linking. - We need to make sure that the indexing is exact enough. Currently () are ignored, so (XXXX)1235 woudl be found when looking for 1235. If you have multiple $0 that could cause issues. Right now, the linking is exact. I am not saying we shouldn't discuss this, but I think it needs more work. I see that some authority fields are hardcoded - a lot of libraries seem to put authority control on other fields as well as Koha allows this in configuration.
Ahh, missed that. Thank you, will fix that. :) I even tested with multiple $0 subfields in place. It seemed to work fine. Although there could be some limiting like integer would be the one Koha uses and if there are characters then we assume it is from other source.
Created attachment 114433 [details] [review] Bug 17511 - Authority linking subfield Bug 17511: Authority linking subfield Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0. This commit will add a system preference AuthoritySubfieldLink which is used to define the linking subfield. The default subfield is set to $9. To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex. TEST PLAN: 1) Add new authority linking to a record. 2) Verify that the authority records ID is on $9 subfield. 3) Check index that authority has one record chosen. 4) Apply the patch. 5) Check AuthoritySubfieldLink to be 9. 6) Repeat steps 1, 2 and 3. 7) Change the preference value 8) Replace zebradb/marc_defs with new files or/and reset ES configurations. 9) Add new authority linking to a record. 10) Verify that the authority records ID is now on the chosen subfield. 11) Check index again that authority has record chosen. Sponsored-by: Koha-Suomi Oy
I modified the old patch slightly, now the auhtority linking subfield can be defined in system preferences. The default value is 9 so nothing changes if you don't want to change it to be something else.
I was thinking about this idea just the other day... If you look at https://www.loc.gov/marc/bibliographic/ecbdcntf.html, you'll see that $0 can also be a URI. That could be pretty cool. It seems too that the $0 is repeatable so that you could have different identifiers... I guess so you could reference VIAF and something else?
(In reply to Katrin Fischer from comment #4) > I think another reason might be that Koha was using $9 when there was no $0 > subfield in MARC21 yet. Bingo. According to https://www.loc.gov/marc/bibliographic/ecbdcntf.html, the $0 was added (for this purpose) in 2007.
(In reply to David Cook from comment #18) > I was thinking about this idea just the other day... > > If you look at https://www.loc.gov/marc/bibliographic/ecbdcntf.html, you'll > see that $0 can also be a URI. That could be pretty cool. > > It seems too that the $0 is repeatable so that you could have different > identifiers... I guess so you could reference VIAF and something else? We could index a URI, so that Koha's bib to auth link search would still work... We could actually build in a RDF dereferencing system eventually.
(In reply to David Cook from comment #20) > (In reply to David Cook from comment #18) > > I was thinking about this idea just the other day... > > > > If you look at https://www.loc.gov/marc/bibliographic/ecbdcntf.html, you'll > > see that $0 can also be a URI. That could be pretty cool. > > > > It seems too that the $0 is repeatable so that you could have different > > identifiers... I guess so you could reference VIAF and something else? > > We could index a URI, so that Koha's bib to auth link search would still > work... > > We could actually build in a RDF dereferencing system eventually. Do you mean putting Koha's internal link to authority linking subfield? We are using national metadata repository and mainly the URI in $0 subfield has a link to that. It might be a good idea that Koha would use URI in authority linking subfield also. We are facing problem with this $9 subfield because it is used in our national metadata repository for other things and it should not change if we are pulling data to Koha. This is why it would be nice to choose which subfield is used for the authority linking.
Created attachment 114474 [details] [review] Bug 17511 - Authority linking subfield The patch had wrong author information, replaced with my own.
(In reply to Johanna Räisä from comment #21) > Do you mean putting Koha's internal link to authority linking subfield? > I'm not 100% sure what you mean, but I think so, yes. > We are using national metadata repository and mainly the URI in $0 subfield > has a link to that. It might be a good idea that Koha would use URI in > authority linking subfield also. > Yeah that sounds good to me. So you wouldn't be storing authority records in Koha? > We are facing problem with this $9 subfield because it is used in our > national metadata repository for other things and it should not change if we > are pulling data to Koha. This is why it would be nice to choose which > subfield is used for the authority linking. Hmm, how are you pulling the authority data to Koha?
(In reply to David Cook from comment #23) > (In reply to Johanna Räisä from comment #21) > > Do you mean putting Koha's internal link to authority linking subfield? > > > > I'm not 100% sure what you mean, but I think so, yes. I meant that the authid would be changed to URI and saved that instead of integer value. > > > We are using national metadata repository and mainly the URI in $0 subfield > > has a link to that. It might be a good idea that Koha would use URI in > > authority linking subfield also. > > > > Yeah that sounds good to me. So you wouldn't be storing authority records in > Koha? No we won't. > > > We are facing problem with this $9 subfield because it is used in our > > national metadata repository for other things and it should not change if we > > are pulling data to Koha. This is why it would be nice to choose which > > subfield is used for the authority linking. > > Hmm, how are you pulling the authority data to Koha? We are making a plugin which fetches the authority via SRU or REST API while writing it to the field. We have already a similar approach to get vocabularies https://github.com/KohaSuomi/Koha/commit/9dec82f3c28b41946f8b301dd9200cc36e92cd35
(In reply to Johanna Räisä from comment #24) > (In reply to David Cook from comment #23) > > (In reply to Johanna Räisä from comment #21) > > > Do you mean putting Koha's internal link to authority linking subfield? > > > > > > > I'm not 100% sure what you mean, but I think so, yes. > > I meant that the authid would be changed to URI and saved that instead of > integer value. > Sounds good to me. The only thing to keep in mind is that the $9 is currently indexed into the Koha-Auth-Number:w index which does mean that it's normalized and tokenized. This could cause problems if there is an unencoded space in the URI for instance. It might be wise for us to start using a "p" register instead of a "w" register, and to start doing "exact" searches when querying this index... At least if you're using Zebra. I don't know about Elasticsearch... > > Yeah that sounds good to me. So you wouldn't be storing authority records in > > Koha? > > No we won't. > Interesting! Very interesting! > We are making a plugin which fetches the authority via SRU or REST API while > writing it to the field. We have already a similar approach to get > vocabularies > https://github.com/KohaSuomi/Koha/commit/ > 9dec82f3c28b41946f8b301dd9200cc36e92cd35 Sounds interesting. I'd like to hear more as you progress. When you say that the plugin fetches the authority, do you mean that it fetches it, extracts the relevant data, inserts it into the Koha bibliographic record, and then discards the authority record? I think that's such a great idea.
(In reply to Johanna Räisä from comment #16) > TEST PLAN: > 1) Add new authority linking to a record. > 2) Verify that the authority records ID is on $9 subfield. > 3) Check index that authority has one record chosen. > 4) Apply the patch. > 5) Check AuthoritySubfieldLink to be 9. > 6) Repeat steps 1, 2 and 3. > 7) Change the preference value > 8) Replace zebradb/marc_defs with new files or/and reset ES configurations. > 9) Add new authority linking to a record. > 10) Verify that the authority records ID is now on the chosen subfield. > 11) Check index again that authority has record chosen. > Could you elaborate on Step 8? I think it would be wise to add an additional example patch (marked with something like DONOTPUSH) with your desired configuration for etc/zebradb/marc_defs/marc21/biblios/biblio-zebra-indexdefs.xsl That'll make it easier for us to test. Otherwise, the tester and QA have to guess at the correct configuration. I think that this sounds like a good change though. Wise to keep the $9 for default, and maybe we can announce a deprecation in favour of moving to $0 at some point...
Created attachment 114918 [details] [review] Bug 17511 - Authority linking subfield Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0. This commit will add a system preference AuthoritySubfieldLink which is used to define the linking subfield. The default subfield is set to $9. To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex. TEST PLAN: 1) Add new authority linking to a record. 2) Verify that the authority records ID is on $9 subfield. 3) Check index that authority has one record chosen. 4) Apply the patch. 5) Check AuthoritySubfieldLink to be 9. 6) Repeat steps 1, 2 and 3. 7) Change the preference value to 0 8) Replace zebra defs and elastic mapping file with the commit sample files for testing. 9) Add new authority linking to a record. 10) Verify that the authority records ID is now on the $0 subfield. 11) Check index again that authority has record chosen. Sponsored-by: Koha-Suomi Oy
(In reply to Johanna Räisä from comment #27) > Created attachment 114918 [details] [review] [review] > Bug 17511 - Authority linking subfield > > Koha uses $9 subfield for linking authority records. MARC21 format says the > subfield should be $0. > This commit will add a system preference AuthoritySubfieldLink which is used > to define the linking subfield. > The default subfield is set to $9. > > To change the old linkings run batchAuthorityLinking.pl, modify index > mappings and reindex. > > TEST PLAN: > 1) Add new authority linking to a record. > 2) Verify that the authority records ID is on $9 subfield. > 3) Check index that authority has one record chosen. > 4) Apply the patch. > 5) Check AuthoritySubfieldLink to be 9. > 6) Repeat steps 1, 2 and 3. > 7) Change the preference value to 0 > 8) Replace zebra defs and elastic mapping file with the commit sample files > for testing. > 9) Add new authority linking to a record. > 10) Verify that the authority records ID is now on the $0 subfield. > 11) Check index again that authority has record chosen. > > Sponsored-by: Koha-Suomi Oy I added only indexing file samples to this patch. Have to think how the URI would work with indexing.
(In reply to Johanna Räisä from comment #28) > I added only indexing file samples to this patch. My apologies, Johanna. I meant that the indexing file samples should be in a separate commit/patch and that the commit message should include DONOTPUSH at the start of the commit message. The patch you've uploaded which includes both the code changes and the samples is problematic for the release manager and release maintainers. They can't just ignore the patch with the samples. Could you split up the patch and re-upload?
(In reply to Johanna Räisä from comment #28) > Have to think how the URI would work with indexing. I have been thinking about the same thing. At a glance, with Zebra we're using Koha-Auth-Number with a "w" register, which is fine for integers, but that probably won't work very well with URIs, since it's for storing "words". It's normalized and tokenized. That will lead to problematic indexing. We could use a "p" register which is normalized and not tokenized but it's possible for 2 different URIs to be normalized into the same data, so that's not ideal either. Perhaps we should use a "0" register which is not normalized and not tokenized. To use that we'd need to use the 4=3 structure attribute and I don't think we'd be using it as the Bib-1 spec says it should be used (https://www.loc.gov/z3950/agency/bib1.html), but I think it would work with Zebra. I'd have to try it out though. Technically, the is a special "u" register for URLs, but we also need to optionally store integers, so that won't work either. I don't use Elasticsearch, so I can't speak to it.
(In reply to David Cook from comment #29) > (In reply to Johanna Räisä from comment #28) > > I added only indexing file samples to this patch. > > My apologies, Johanna. I meant that the indexing file samples should be in a > separate commit/patch and that the commit message should include DONOTPUSH > at the start of the commit message. The patch you've uploaded which includes > both the code changes and the samples is problematic for the release manager > and release maintainers. They can't just ignore the patch with the samples. > > Could you split up the patch and re-upload? Oh, you ment that. Sure, I will split them.
Created attachment 115002 [details] [review] Bug 17511 - Authority linking subfield Koha uses $9 subfield for linking authority records. MARC21 format says the subfield should be $0. This commit will add a system preference AuthoritySubfieldLink which is used to define the linking subfield. The default subfield is set to $9. To change the old linkings run batchAuthorityLinking.pl, modify index mappings and reindex. TEST PLAN: 1) Add new authority linking to a record. 2) Verify that the authority records ID is on $9 subfield. 3) Check index that authority has one record chosen. 4) Apply the patch. 5) Check AuthoritySubfieldLink to be 9. 6) Repeat steps 1, 2 and 3. 7) Change the preference value to 0 8) Replace zebra defs and elastic mapping file with the sample files from second patch for testing. 9) Add new authority linking to a record. 10) Verify that the authority records ID is now on the $0 subfield. 11) Check index again that authority has record chosen. Sponsored-by: Koha-Suomi Oy
Created attachment 115003 [details] [review] Bug 17511: (DONOTPUSH) Indexing sample files These are the indexing sample files for testing AuthoritySubfieldLink systempreference.
(In reply to David Cook from comment #30) > (In reply to Johanna Räisä from comment #28) > > Have to think how the URI would work with indexing. > > I have been thinking about the same thing. > > At a glance, with Zebra we're using Koha-Auth-Number with a "w" register, > which is fine for integers, but that probably won't work very well with > URIs, since it's for storing "words". It's normalized and tokenized. That > will lead to problematic indexing. > > We could use a "p" register which is normalized and not tokenized but it's > possible for 2 different URIs to be normalized into the same data, so that's > not ideal either. > > Perhaps we should use a "0" register which is not normalized and not > tokenized. To use that we'd need to use the 4=3 structure attribute and I > don't think we'd be using it as the Bib-1 spec says it should be used > (https://www.loc.gov/z3950/agency/bib1.html), but I think it would work with > Zebra. I'd have to try it out though. This sounds good to me, although I have little experience configuring Zebra. > > Technically, the is a special "u" register for URLs, but we also need to > optionally store integers, so that won't work either. > > I don't use Elasticsearch, so I can't speak to it. I can look Elasticsearch. I have work with it more than I have worked with Zebra. Maybe we should make another bug for this? At least I think this is another enhancement.
I'm intrigued by this one but super busy at the moment, unfortunately.
Created attachment 116405 [details] [review] Bug 17511: (DONOTPUSH) Indexing sample files These are the indexing sample files for testing AuthoritySubfieldLink systempreference. Signed-off-by: Frank Hansen <frank.hansen@ub.lu.se> This patch works fine in elasticsearch at least. Great job!
It would be a good idea to remove the last mentions of $0 from the patch, and also from the bug summary, since this isn't at all a $0 implementation. $0 is a repeatable subfield, with specified contents of "(MARC org code from 003 in an authority record)control number of an authority record", which isn't what this is doing.
(In reply to Phil Ringnalda from comment #37) > It would be a good idea to remove the last mentions of $0 from the patch, > and also from the bug summary, since this isn't at all a $0 implementation. > $0 is a repeatable subfield, with specified contents of "(MARC org code from > 003 in an authority record)control number of an authority record", which > isn't what this is doing. I haven't reviewed these patches in a while, so I can't speak to the first part of that. However, note that $0 is more complex than "(MARC org code from 003 in an authority record)control number of an authority record". See "$0 - Authority record control number or standard number" at the following URL for more details: https://www.loc.gov/marc/bibliographic/ecbdcntf.html
To add a wrinkle to this discussion. OCLC is using $0 in authority records, perhaps as intended with linking information. An example authority: Conduct of life. $2 fast $0 (OCoLC)fst00874563 In this case, a library wants to retain this subfield as a reference point. It would cause a conflict if a library is using $0 for internal linking and brought in a record which contains reference data in this field for a different system.