Bug 22972 - Proposal for enriching the bibliographic records with standard identifiers from authority data
Summary: Proposal for enriching the bibliographic records with standard identifiers fr...
Status: Failed QA
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low enhancement
Assignee: Julian Maurice
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-05-23 10:08 UTC by Janusz Kaczmarek
Modified: 2024-10-25 07:48 UTC (History)
11 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 22972: Add another source of authority data to copy in biblio record (7.82 KB, patch)
2023-09-19 11:47 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 22972: Modify MARC21 XSLT to include link to external data ($1) (7.47 KB, patch)
2023-09-19 11:47 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 22972: Update DBIC schema (1.50 KB, patch)
2023-09-19 11:47 UTC, Julian Maurice
Details | Diff | Splinter Review
Bug 22972: Add another source of authority data to copy in biblio record (6.64 KB, patch)
2024-08-16 09:03 UTC, Baptiste Wojtkowski (bwoj)
Details | Diff | Splinter Review
Bug 22972: Modify MARC21 XSLT to include link to external data ($1) (7.46 KB, patch)
2024-08-16 09:03 UTC, Baptiste Wojtkowski (bwoj)
Details | Diff | Splinter Review
Bug 22972: Update DBIC schema (1.49 KB, patch)
2024-08-16 09:03 UTC, Baptiste Wojtkowski (bwoj)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Janusz Kaczmarek 2019-05-23 10:08:02 UTC
Thinking in terms of linked data, it would be of profit to have as much of standard identifiers in the bibliographic data produced by the library as possible. 

Libraries use authority data coming from official sources (like LoC, SUDOC, etc.) or enrich their locally created authority records with standard identifiers (e.g. viafId, ISNI, Wikipedia).  This information is linked to the bibliographic record by the Koha-specific subfield $9 which is illegal in terms of MARC 21 and does not mean anything outside of the local Koha ecosystem.  A record moved out of a local Koha catalogue (downloaded, exported) loses the linked-data connection to other objects (like authors, subjects etc.), which are then identified only by strings. 

To make this connection between bibliographic access point and objects identified by universal identifiers more permanent outside of the local environment, the standard MARC 21 $0 subfield could be used. (UNIMARC has to be examined - $3 subfield?)

The required changes would include at least: C4::AuthoritiesMarc::merge function, the authority plugin, some new system preferences, optionally display templates and also a script to update the existing biblio records. 

New possible system preferences needed:
- switch the feature on/off 
- source of the standard IDs (possibly different for different biblio fields?) (standard in MARC 21 authority: 024, but may be also 010/035) (a list)
- how many ids should be copied (= how may $0 added): only the first found or all of them
- if an extra link/icon should be generated in display (OPAC / librarian)

Please, share your comments about this idea.
Comment 1 Marcel de Rooy 2020-04-02 08:11:30 UTC
For linked data we probably want to use $1 to add links to standard vocabularies like Wikidata, VIAF, ULAN etc.
Comment 2 Janusz Kaczmarek 2020-04-07 20:40:41 UTC
Yes, I agree, definitely.  The (relatively) new $1 should be used for identifying Real Would Objects.
Comment 3 Paul Poulain 2021-11-24 11:30:25 UTC
One of our libraries is interested by this feature, they asked me to have a look at this ticket.

Some explanations (from history) : I'm the developer who wrote the MARC management in Koha (in 2003...)
The $9 was chosen to store a link between biblio & authority because it was for local use (as any field/subfield with a 9).
In UNIMARC, the $3 was a possible field (as it contains the linkage), in MARC21 there was no subfield for that. But anyway, I thought (and still think) it was better to keep the $3 as it is and use a local field because it stores a local information (the authid of the authority in the local [Koha] system) In France, the SUDOC (national academic catalogue) makes an extensive usage of the $3, we must preserve it.

Otherwise :

That's a very good idea to add the wikidata reference (or any other one) and your specification seems OK, I have a couple of comment though:

- switch the feature on/off
=> not sure it's needed, if the next syspref is empty, it's OFF ;)

- source of the standard IDs (possibly different for different biblio fields?) (standard in MARC 21 authority: 024, but may be also 010/035) (a list)
=> I agree, it must not be at a global level. But counter-proposal : define that in the authority level. In the authority definition, we define which authority field must be copied in the biblio field [for example, corporate name is copied from the 110 of the authority]. Shouldn't we have another option to say "024$a" is copied in the $1 of the biblio ? [the $1 is possible for UNIMARC, it's an unused field]

- how many ids should be copied (= how may $0 added): only the first found or all of them
=> I would say "copy only one". Of maybe add another option in the authority definition to say "add the 1st where source/$2=[_____]" [$2 is valid for UNIMARC too]

- if an extra link/icon should be generated in display (OPAC / librarian)
=> I think we just need to improve the XSLT for that.

Very interesting proposal though ! I'll continue talking with the library to see if they can sponsor the dev
Comment 4 Marcel de Rooy 2022-10-06 13:36:32 UTC
Thinking along these lines about it:

Data looks like
024 7# - OTHER STANDARD IDENTIFIER
    Standard number or code: https://id.rijksmuseum.nl/31018715
    Source of number or code: uri
024 7# - OTHER STANDARD IDENTIFIER
    Real World Object URI: https://rkd.nl/explore/artists/15189
    Source of number or code: rkda
024 7# - OTHER STANDARD IDENTIFIER
    Real World Object URI: http://www.wikidata.org/entity/Q454568
    Source of number or code: wikidata

Add a pref to know which field numbers to copy (024 for MARC21).
Copy $1 from auth to $1 at biblio side. (Repeatable)
Leave #2 alone. It is something different.
In the above case we need to decide on the best way to show all (3) additional URIs on the detail page that come with this author (or another auth type).
Comment 5 Katrin Fischer 2022-11-14 21:24:23 UTC
For MARC21 I think identifiers for $0 are often found in 001/035:
https://www.loc.gov/marc/authority/ad035.html
Comment 6 bruno.forment@orpheusinstituut.be 2022-11-15 08:48:30 UTC
(In reply to Katrin Fischer from comment #5)
> For MARC21 I think identifiers for $0 are often found in 001/035:
> https://www.loc.gov/marc/authority/ad035.html

Those identifiers work only on the level of the complete record, not with respect to the various authorities (personal, geographic, topical terms etc.).
Comment 7 Katrin Fischer 2022-11-15 10:17:47 UTC
(In reply to bruno.forment@orpheusinstituut.be from comment #6)
> (In reply to Katrin Fischer from comment #5)
> > For MARC21 I think identifiers for $0 are often found in 001/035:
> > https://www.loc.gov/marc/authority/ad035.html
> 
> Those identifiers work only on the level of the complete record, not with
> respect to the various authorities (personal, geographic, topical terms
> etc.).

Hi Brono, I meant the 035 in authorities, not 035 in the bibliographic record. In our case for example one of the 035 of the authority has the ID from the national authority database for this person/subject/topical term.
Comment 8 Julian Maurice 2023-09-19 11:47:42 UTC
Created attachment 155890 [details] [review]
Bug 22972: Add another source of authority data to copy in biblio record

This patch allows to copy an authority subfield into a biblio record
subfield ($1) when cataloging a biblio record and using the authority
finder. This can be used to copy standard authority identifiers to the
biblio record

The subfield is configurable for each authority type.

If multiple fields or subfields exist in the authority, they are all
copied to the biblio record in separate $1 subfields

Test plan (authority types and fields are for MARC21, but this work in
UNIMARC too):
1. Edit PERSO_NAME autority type and set "Authority subfield to copy in
   $1" to: 024$1
2. Check that you have (or create) a PERSO_NAME authority of this type
   with several 024$1
3. Check that in the biblio framework 700$1 is defined and visible
4. In biblio field 700, choose the above authority. Check that the
   024$1 content is present in 700$1

Sponsored-by: Orpheus Instituut
Sponsored-by: Rijksmuseum Amsterdam
Sponsored-by: Saxion Hogeschool
Sponsored-by: Breda University of Applied Sciences
Sponsored-by: FotoMuseum Antwerpen
Sponsored-by: ModeMuseum Antwerpen
Sponsored-by: DIVA Museum for Diamonds, Jewellery and Silver
Sponsored-by: Koninklijk Conservatorium Brussel
Comment 9 Julian Maurice 2023-09-19 11:47:46 UTC
Created attachment 155891 [details] [review]
Bug 22972: Modify MARC21 XSLT to include link to external data ($1)
Comment 10 Julian Maurice 2023-09-19 11:47:49 UTC
Created attachment 155892 [details] [review]
Bug 22972: Update DBIC schema
Comment 11 Marcel de Rooy 2023-09-19 11:55:56 UTC
Hi Julian,
How does this work together with pref AuthorityMergeMode ?
If you are in strict mode, the merge is overwriting or removing subfields.
Comment 12 Julian Maurice 2023-09-19 12:18:04 UTC
Data is not copied to the biblio field while merging authorities. I haven't tested but I suppose that if you are in strict mode, the biblio field's $1 is removed.
Comment 13 Marcel de Rooy 2023-09-19 12:42:44 UTC
(In reply to Julian Maurice from comment #12)
> Data is not copied to the biblio field while merging authorities. I haven't
> tested but I suppose that if you are in strict mode, the biblio field's $1
> is removed.

Yes, thats my point. It would be a temporary enrichment. But note that 99% is on loose mode luckily. (Checked HEA numbers.) Should this be noted somewhere?
Comment 14 Fridolin Somers 2024-01-09 08:25:08 UTC
diff --git a/installer/data/mysql/atomicupdate/bug-22972.pl b/installer/data/mysql/atomicupdate/bug-22972.pl
new file mode 100644

Atomic updates need perm 755
Comment 15 Thibault Keromnès 2024-01-22 09:50:52 UTC
Patch doesn't patch apply : 
CONFLICT (content): Merge conflict in koha-tmpl/intranet-tmpl/prog/en/modules/admin/authtypes.tt
Comment 16 Baptiste Wojtkowski (bwoj) 2024-08-16 09:03:03 UTC
Created attachment 170443 [details] [review]
Bug 22972: Add another source of authority data to copy in biblio record

This patch allows to copy an authority subfield into a biblio record
subfield ($1) when cataloging a biblio record and using the authority
finder. This can be used to copy standard authority identifiers to the
biblio record

The subfield is configurable for each authority type.

If multiple fields or subfields exist in the authority, they are all
copied to the biblio record in separate $1 subfields

Test plan (authority types and fields are for MARC21, but this work in
UNIMARC too):
1. Edit PERSO_NAME autority type and set "Authority subfield to copy in
   $1" to: 024$1
2. Check that you have (or create) a PERSO_NAME authority of this type
   with several 024$1
3. Check that in the biblio framework 700$1 is defined and visible
4. In biblio field 700, choose the above authority. Check that the
   024$1 content is present in 700$1

Sponsored-by: Orpheus Instituut
Sponsored-by: Rijksmuseum Amsterdam
Sponsored-by: Saxion Hogeschool
Sponsored-by: Breda University of Applied Sciences
Sponsored-by: FotoMuseum Antwerpen
Sponsored-by: ModeMuseum Antwerpen
Sponsored-by: DIVA Museum for Diamonds, Jewellery and Silver
Sponsored-by: Koninklijk Conservatorium Brussel
Comment 17 Baptiste Wojtkowski (bwoj) 2024-08-16 09:03:07 UTC
Created attachment 170444 [details] [review]
Bug 22972: Modify MARC21 XSLT to include link to external data ($1)
Comment 18 Baptiste Wojtkowski (bwoj) 2024-08-16 09:03:10 UTC
Created attachment 170445 [details] [review]
Bug 22972: Update DBIC schema
Comment 19 Baptiste Wojtkowski (bwoj) 2024-08-16 09:03:54 UTC
Rebased on main
Comment 20 Marcel de Rooy 2024-08-16 09:27:52 UTC
Thx for rebasing.
The field name subfield_to_report_in_1 needs to be improved imo. This is not very clear.
Comment 21 Janusz Kaczmarek 2024-09-24 10:02:52 UTC
Thank you, Baptiste, for supplying this patch. We tested it: it works fine for adding $1 (or any other subfield indicated in configuration) in the standard editor. 

What doesn't work:

1. [advanced editor] In the advanced editor, if a authority record has more than one 024 field, all occurrences of the indicated subfield are concatenated into a single string and placed in a single $1 subfield.

2. [update/merge] Adding the 024 field with the subfield indicated in the configuration to the authority record does not add subfield $1 to the heading field in bibliographic records linked with the authority record in question.

3. [link] In a situation where authority and bibliographic records have been loaded into the system from an external source, and then the linking process has been started (link_bibs_to_authorities.pl), the 024 $1 subfields (or other subfields indicated in the configuration, respectively) are not added as $1 subfields in the corresponding headings of the bibliographic records. 

There is also the question of how Koha should behave in the case of authority records with the 024 $1 subfields already present in the database and linked to bibliographic records - wouldn't we need a script to rewrite the headings and add the $1 subfields.

And a final note: displaying information from the subfield $1 in the bibliographic record view seems very unergonomic to us, especially for URIs outside the list hardcoded in the XSLT (currently: viaf and wikidata), such as ISNI. We would suggest integrating this information into the functionality introduced by bug 29948, as a possible follow-up. 

Would you be ready to work further on 1, 2, 3? And what do you think about final remarks...
Comment 22 Marcel de Rooy 2024-10-25 07:48:41 UTC
(In reply to Janusz Kaczmarek from comment #21)

> 1. [advanced editor] In the advanced editor, if a authority record has more
> than one 024 field, all occurrences of the indicated subfield are
> concatenated into a single string and placed in a single $1 subfield.
> 
> 2. [update/merge] Adding the 024 field with the subfield indicated in the
> configuration to the authority record does not add subfield $1 to the
> heading field in bibliographic records linked with the authority record in
> question.
> 
> 3. [link] In a situation where authority and bibliographic records have been
> loaded into the system from an external source, and then the linking process
> has been started (link_bibs_to_authorities.pl), the 024 $1 subfields (or
> other subfields indicated in the configuration, respectively) are not added
> as $1 subfields in the corresponding headings of the bibliographic records. 
> 
> There is also the question of how Koha should behave in the case of
> authority records with the 024 $1 subfields already present in the database
> and linked to bibliographic records - wouldn't we need a script to rewrite
> the headings and add the $1 subfields.
> 
> And a final note: displaying information from the subfield $1 in the
> bibliographic record view seems very unergonomic to us, especially for URIs
> outside the list hardcoded in the XSLT (currently: viaf and wikidata), such
> as ISNI. We would suggest integrating this information into the
> functionality introduced by bug 29948, as a possible follow-up. 

Thanks Janusz for this detailed list of pending issues here.