Bug 28491 - Field 003 in authority records not updated after import
Summary: Field 003 in authority records not updated after import
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Authority data support (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Bugs List
QA Contact: Kyle M Hall
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2021-05-31 12:06 UTC by Thomas Klausner
Modified: 2024-02-06 17:59 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 28491: Only whitespace changes for AddAuthority (4.33 KB, patch)
2021-06-21 08:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 28491: Always update field 003 in AddAuthority (MARC21) (1.73 KB, patch)
2021-06-21 08:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 28491: Add unit test (3.46 KB, patch)
2021-06-21 08:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review
Bug 28491: Only whitespace changes for AddAuthority (4.29 KB, patch)
2021-06-22 16:56 UTC, Hakam Almotlak
Details | Diff | Splinter Review
Bug 28491: Add unit test (3.46 KB, patch)
2021-06-22 17:01 UTC, Hakam Almotlak
Details | Diff | Splinter Review
Bug 28491: Always update field 003 in AddAuthority (MARC21) (1.69 KB, patch)
2021-06-22 17:02 UTC, Hakam Almotlak
Details | Diff | Splinter Review
Bug 20206: Find barcodes with more or less leading zeroes (3.10 KB, patch)
2021-06-22 17:29 UTC, Hakam Almotlak
Details | Diff | Splinter Review
Bug 20206: Find barcodes with more or less leading zeroes (3.10 KB, patch)
2021-06-22 17:30 UTC, Hakam Almotlak
Details | Diff | Splinter Review
Bug 28491: Always update field 003 in AddAuthority (MARC21) (1.73 KB, patch)
2021-06-22 17:31 UTC, Hakam Almotlak
Details | Diff | Splinter Review
Bug 28491: Always update field 003 in AddAuthority (MARC21) (1.73 KB, patch)
2021-06-22 17:32 UTC, Hakam Almotlak
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Thomas Klausner 2021-05-31 12:06:41 UTC
I import some authority data from e.g. dnb.de, which contains:

<controlfield tag="001">11915725X</controlfield>
<controlfield tag="003">DE-101</controlfield>


After the import, the local authority data looks like

<controlfield tag="001">35</controlfield>
<controlfield tag="003">DE-101</controlfield>


So 001 now contains the internal Koha id (good), but 003 still points to DE-101 (i.e. dnb.de), which I think is bad, because if I look up the ID '35' in the linked DB DE-101, I will now get a bad result.

I assume that 003 should be set to my MARCOrgCode


Looking at the code, I find (in C4::AuthoritiesMarc, AddAuthority, 568ff)

if (!$record->field('003')) {
   $record->insert_fields_ordered(
      MARC::Field->new('003', $marcorgcode),
   );
}

So this behavior seems intentional? If so, I still find it very unlogical and buggy. I assume that both 001 and 003 should be overwritten with data pointing to the current Koha instance, and the original 001/003 "archived" in 035 and/or 040

I could easily provide a patch that changes this behavior, but I'm not sure if this would cause confusion / failing tests / etc.

Maybe add an option to allow a force-overwrite of 003?

Greeting,
Thomas

PS this seems slightly related to #25603
Comment 1 Katrin Fischer 2021-06-03 20:28:03 UTC
I am adding Marcel as I know he has been working some on the code we talk about here. 

My opinion would be to overwrite with MarcOrgCode or empty 003 when we add the Koha number there.
Comment 2 Marcel de Rooy 2021-06-18 09:23:09 UTC
I understand your point. If we change 001, we should change 003 as well. But note that we have a lot of discussion for years going on if we should change 001 at biblio level.
As you mentioned, the 003 is created for a new auth record but not for an import.

LOC: Whenever the number in field 001 is changed, agencies must assure that the MARC code in field 003 applies to the number found in the 001 field.

I am all for changing 003 here as well. But could imagine that others would rather refrain from changing 001?
Comment 3 Katrin Fischer 2021-06-19 11:17:07 UTC
> I am all for changing 003 here as well. But could imagine that others would
> rather refrain from changing 001?

I usually would, but in this case it's already the default behaviour of the authority import script :) So when we already change it, I think changing 003 as well just makes it 'right'.  Bibliographic records should be a separate discussion I think.
Comment 4 Marcel de Rooy 2021-06-21 08:32:13 UTC
Created attachment 122209 [details] [review]
Bug 28491: Only whitespace changes for AddAuthority

Only fixes indentation at highest level in sub.
Includes warnings from qa tools:
    forbidden pattern: trailing space char (line 600)
    forbidden pattern: trailing space char (line 602)
    forbidden pattern: trailing space char (line 608)
    forbidden pattern: trailing space char (line 613)

Test plan:
Nothing to test really. No compile warnings?

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 5 Marcel de Rooy 2021-06-21 08:32:17 UTC
Created attachment 122210 [details] [review]
Bug 28491: Always update field 003 in AddAuthority (MARC21)

Test plan:
Add or modify a record.
Verify that the 003 always become the branch or system orgcode.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 6 Marcel de Rooy 2021-06-21 08:32:20 UTC
Created attachment 122211 [details] [review]
Bug 28491: Add unit test

Test plan:
Run t/db_dependent/AuthoritiesMarc.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 7 Hakam Almotlak 2021-06-22 16:56:34 UTC Comment hidden (obsolete)
Comment 8 Hakam Almotlak 2021-06-22 17:01:11 UTC
Created attachment 122295 [details] [review]
Bug 28491: Add unit test

Test plan:
Run t/db_dependent/AuthoritiesMarc.t

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: hakam <hakam@inlibro.com>
Comment 9 Hakam Almotlak 2021-06-22 17:02:08 UTC
Created attachment 122296 [details] [review]
Bug 28491: Always update field 003 in AddAuthority (MARC21)

Test plan:
Add or modify a record.
Verify that the 003 always become the branch or system orgcode.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Comment 10 Hakam Almotlak 2021-06-22 17:29:18 UTC Comment hidden (obsolete)
Comment 11 Hakam Almotlak 2021-06-22 17:30:05 UTC Comment hidden (obsolete)
Comment 12 Hakam Almotlak 2021-06-22 17:31:09 UTC
Created attachment 122299 [details] [review]
Bug 28491: Always update field 003 in AddAuthority (MARC21)

Test plan:
Add or modify a record.
Verify that the 003 always become the branch or system orgcode.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: hakam <hakam@inlibro.com>
Comment 13 Hakam Almotlak 2021-06-22 17:32:03 UTC
Created attachment 122300 [details] [review]
Bug 28491: Always update field 003 in AddAuthority (MARC21)

Test plan:
Add or modify a record.
Verify that the 003 always become the branch or system orgcode.

Signed-off-by: Marcel de Rooy <m.de.rooy@rijksmuseum.nl>
Signed-off-by: hakam <hakam@inlibro.com>
Comment 14 Marcel de Rooy 2021-06-23 07:55:49 UTC Comment hidden (obsolete)
Comment 15 Marcel de Rooy 2021-06-23 11:16:55 UTC
Thx Hakam for your signoff btw
Comment 16 Nick Clemens 2021-08-09 13:11:51 UTC
While this is all technically correct according to MARC standards, I do think it will have an impact on library workflows.

Some libraries use the 003 for determining the source of the record, automatically updating the field is going to be a change.

perhaps a checkbox, checked by default, during import?
Comment 17 Marcel de Rooy 2021-08-09 13:26:46 UTC
I do not mind obsoleting this development, if it is just too hurtful :)

We found a comfortable in-between solution: the Discussion status, hahaha
Comment 18 Katrin Fischer 2021-08-09 14:00:06 UTC
They can still determine the source using 035 - is that not enough?

001 and 003 are a couple. 

At the moment we risk that we generate "false" couples that might match real records in the original source - that could lead to issues on importing and matching for example.

I really think this is a bug. And it should be possible to make a permanent change in that behaviour - a checkbox that might sometimes be forgotten is too dangerous.
Comment 19 Martin Renvoize 2021-08-09 14:50:02 UTC
(In reply to Katrin Fischer from comment #18)
> They can still determine the source using 035 - is that not enough?
> 
> 001 and 003 are a couple. 
> 
> At the moment we risk that we generate "false" couples that might match real
> records in the original source - that could lead to issues on importing and
> matching for example.
> 
> I really think this is a bug. And it should be possible to make a permanent
> change in that behaviour - a checkbox that might sometimes be forgotten is
> too dangerous.

I think the more "natural" pref for me would be to keep 001+003 or to change both. This is certainly my understanding of the specs.
Comment 20 Katrin Fischer 2021-08-09 15:10:52 UTC
I assumed we already moved to 035 earlier, but not sure now. 
Not sure when I get to test this out, does someone know?

I think to be fully compliant it would be:

Copy 001 and 003 as "(003)001" to 035.
Update 001 to internal ID and 003 to MARCOrgCode.
Comment 21 Heather 2021-08-11 15:31:43 UTC
We are on 20.05 and authority records that we import into Koha from OCLC's Connexion Client via the gateway have the OCLC record's ARN/001 number completely deleted and replaced and the 003 is: OCoLC
...which isn't correct--for us, the 003 should then be CSfMM.

It would be great if the data in the OCLC record's ARN/001 could be *correctly* mapped into the 035 field.  For an OCLC record number, there could possible be a prefix like with OCLC bibliographic record numbers, but I'm not familiar with the formats for aut record numbers, since we never have them preserved.:)  For us, the important numbers are in the 010 field, the Library of Congress Control Number, but this is U.S.-based.

For *bibliographic* 035 fields, the MARC code of the originator of the control number can be enclosed in parentheses preceding the number, e.g.:
035 ## $a (CaOTULAS)41063988

So perhaps someone could research if it's valid to format an aut 035 similarly, e.g.:
035 ## $a (DE-101)11915725X
Comment 22 Martin Renvoize 2021-08-12 08:59:29 UTC
Looks pretty much like that's the case to me: https://www.loc.gov/marc/authority/ad035.html
Comment 23 Martin Renvoize 2021-08-12 09:15:12 UTC
Do you understand how subfield z vs subfield a work here.. my interpretation is that a = 'current' (i.e what we're replacing with) and z = 'cancelled'.

So am I right in thinking.. if we replace 001 + 003 on import we should add a 035 with a = (new003)new001 and z (original003)original001.

As 035 is repeatable I can see that meaning we get an audit trail one could trace back through using the two subfields to get the order of actions... but I'm no librarian so I may to totally misinterpreting that.
Comment 24 Katrin Fischer 2021-08-12 09:22:08 UTC
I haven't seen $z used like this tbh. I've only seen the numbers moved directly to 035$a around here and no use of $z at all. From the docs:

https://www.loc.gov/marc/bibliographic/bd035.html
For interchange purposes, documentation of the structure and use of the system control number must be provided to exchange partners by the originating organization. Each valid system control number and any related canceled/invalid control number are contained in a separate 035 field. 

So as our number is still valid at its source, I think $a is correct.

I believe $z might be used if records are merged in their source catalog like the example suggests for WorldCat? A number that really no longer exists, no longer is in use at the source.
Comment 25 Heather 2021-08-12 13:40:51 UTC
My understanding of the use of $z in many fields, including the 035, is the same as Katrin's.  A number *that is still valid* in use in another system can't be placed in a $z, and it hasn't been canceled in that other system.  It is used exactly in that way in WorldCat--when records are merged, the ID numbers of the subordinated & replaced records are placed in a $z of an 035.

A $z is also used in a similar way in bib records in the 020 ISBN field, for example, if a lot of books have a misprinted and incorrect ISBN number, it will be placed in a $z with the correct number in a $a.  But some people may still be seeing and using the incorrect number, thus its presence in the record in the $z.

So I think, too, that other system's valid ID numbers should be in a $a in 035s.
Comment 26 Phil Ringnalda 2023-05-20 00:43:44 UTC
(In reply to Martin Renvoize from comment #23)
> Do you understand how subfield z vs subfield a work here.. my interpretation
> is that a = 'current' (i.e what we're replacing with) and z = 'cancelled'.
> 
> So am I right in thinking.. if we replace 001 + 003 on import we should add
> a 035 with a = (new003)new001 and z (original003)original001.

That idea misses the text in https://www.loc.gov/marc/bibliographic/bd035.html saying "Control number for the record in a system other than the one whose control number is contained in field 001 (Control Number), field 010 (Library of Congress Control Number), or field 016 (National Bibliographic Agency Control Number)."

Since ours is in 001, we shouldn't be putting it in 035, never mind the part where $z should indeed mean "the organization in () has said that this number is canceled or invalid" rather than "I'm personally not currently using this number in 001."

But, neither of those mean we can't push ahead with misusing 035, since the entire idea requires misuse.

If the original 003 is a national library and the original 001 is in either 010 or 016, then putting it in 035 is also wrong. However, the existence of an 010 doesn't mean a record came from the LC, and unless we delete existing 035s before adding ours, the existence of an 035 $a(OCoLC) doesn't mean it came from OCLC. In fact, for a name heading first created by a NACO cataloger using OCLC, the OCLC record has an 010 but no 035 because OCLC's control number is in 001, and the LC record has an OCLC 035 from when LC imported it.

And there's no meaning to the order of fields, and no 035 subfield which in any way says "this is the one that means something about where a Koha user imported this record from" so we would either have to use a very fragile system of saying "the source of the record is in the (first|last) 035, unless one of your employees or an outside contractor doing authority control didn't know that special fact and put one (before|after) it" or be overly-aggressive and delete every existing 035 before adding our One True 035.

Or, we could do what the spec really wants, and define 935 $a(original003)original001 as being (within a Koha instance) the source from which we imported a record, and then people would only have to look in two places: if 935 exists, it's where it came from, if not and 003 is not your marcorgcode (or, not any of your marcorgcodes if you have them per-library) then it's an old import and the 003 is where it came from, if not then 003 is your marcorgcode and you're where it came from. We would have to clobber any existing 935, whether it was one of ours coming from an import from another Koha instance or one from elsewhere with a different meaning, but hey, that's how locally-defined works.
Comment 27 David Cook 2023-05-21 23:15:59 UTC
"...The MARC code identifying whose system control number is present in field 001 is contained in field 003 (Control Number Identifier). An organization using a record of another organization may move the incoming control number from field 001 (and the control number identifier from field 003) to field 035 (System Control Number), 010 (Library of Congress Control Number), or 016 (National Bibliographic Agency Control Number), as appropriate, and place its own system control number in field 001 (and its control number identifier in field 003)."

https://www.loc.gov/marc/authority/ad001.html
Comment 28 Heather 2023-05-22 16:33:54 UTC
IMO, there really is no way to "misuse" an 035 field--a number in the 035 field is simply a number that is not the current/valid system control number, but is a number to be retained.  (Retaining control numbers being a very good idea, IMO!)  Putting a number that is valid in all the other systems and catalogs in the universe except for the one that the record is in right now is a perfectly valid and IMO very good use of the 035.  035s sure can mean, "I'm personally not currently using this number in 001."

Also, IMO, there is no way to accurately trace source(s) of records. Records flow through systems, they are shared--that is really the point of cataloging according to standards, so the records can be shared.  IMO.:)

Figuring out which human or process imported a record into a catalog (the source of a record) works great for bibs in Koha--I can take a look a bib in the Advanced Bibliographic Editor and see in the information in the 945 field.  (e.g., "Connexion Client User"--that's me!)  And there are logs.

If there were an Advanced Authority Editor I would probably be able to see the source of any authority record in a 945 field, but that's another bug/development...:)

The MARC21 standard for authority records is that 001/003 work in tandem.  It works great for bibs.  IMO it should work the same for auts.  The source of a bib in Koha is in the locally-defined 945 and works great; it should be the same for auts (and maybe it is?).  If libraries are using the 003 for a source of a record, that is OK *if* it is functionally paired with the record number in the 001, but IMO is duplicating information that *should* be present in a 945 (that should be visible.)  (Maybe there is a way to make a 945 visible in an aut record framework, but I don't see any authority record framework in the MARC frameworks, but, again, probably another bug!)

I wish this had gone ahead as was signed off in 2021:  if a library wants a different number in the 001 then the replaced number in the 001 and the therefore obsolute org code in the 003 is mapped to a $z in the 035, e.g., $z (ReplacedOrgCode)ReplacedNumber And the source of the record could be, as in bibs, in a locally-defined 945 field to serve the functionality for libraries who need the source of the record tracked in the record as is done in bibs.

--h2, who always has *some* sort of cataloging-informed opinion!:)
Comment 29 Esther Melander 2024-02-06 17:59:26 UTC
I agree that moving the 001 control number of an incoming authority record to the 035 needs to happen. Some third party suppliers of authority records are using the 001 for their control number. Once that record is imported, the third party control number is lost. On the next round of authority updates it then becomes impossible to match and overlay those records. If the incoming authority is missing the corresponding 003, then it should just be left off when the number is copied over to the 035. If this copying action is done, then the 003 should be the local organizations code or blank if not defined.

The current work around is to edit the authorities using MarcEdit to move/copy the 001 control number to the 035 before importing. A task that could be pretty tedious when there are thousands of records that need to be imported and overlaid every three months.