Bug 38142

Summary: UNIMARC: Choose heading to use from the authority record in the bibliographic record by language
Product: Koha Reporter: Matthias Meusburger <matthias.meusburger>
Component: CatalogingAssignee: Matthias Meusburger <matthias.meusburger>
Status: Pushed to main --- QA Contact: Martin Renvoize (ashimema) <martin.renvoize>
Severity: enhancement    
Priority: P5 - low CC: f.demians, lucas, m.de.rooy, martin.renvoize, pasi.kallinen
Version: MainKeywords: release-notes-needed
Hardware: All   
OS: All   
GIT URL: Change sponsored?: ---
Patch complexity: --- Documentation contact:
Documentation submission: Text to go in the release notes:
This allows to choose which heading to copy from an authority record into the bibliographic record using a language code recorded in $7 of the authority heading field. This feature only applies to UNIMARC.
Version(s) released in:
25.05.00
Circulation function:
Attachments: Bug 38142: Choose language to report from authority to bibliographic record.
Bug 38142: Choose language to report from authority to bibliographic record.
Bug 38142: Choose language to report from authority to bibliographic record.
Bug 38142: Choose language to report from authority to bibliographic record.
Bug 38142: Choose language to report from authority to bibliographic record.
Bug 38142: Choose language to report from authority to bibliographic record.
Bug 38142, Follow-up: Choose language to report from authority to bibliographic record.
Bug 38142, Follow-up: Choose language to report from authority to bibliographic record.

Description Matthias Meusburger 2024-10-10 15:58:59 UTC
Currently, when reporting authority field to the bibliographic records, the merge function uses the first matching authority field.
    
However, authorities can provide multiple versions of the same field, with different languages.

It would be nice to have a way to select which language to report in the bibliographic record.
Comment 1 Matthias Meusburger 2024-10-11 09:32:09 UTC
Created attachment 172684 [details] [review]
Bug 38142: Choose language to report from authority to bibliographic record.

Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.

However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.

French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d

This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.

When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.

If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.

The $7 in the authority field can be in a short or long form:

 - the short form is 2 characters long. Example: ba
 - the long form is 8 characters long, with the short form in position 5 and 6
   (starting from 1). Example: ba0yba0y

Test plan:
----------

1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable

2) Add an authority with several languages, like this (the order is important):

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba0yba0y
       _aItō

3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
 search for the authority, then Search main heading ($a only): contains Itō)

Note that you can choose which version you want when searching for the
authority, under the "Get it!" column

For this example, choose the latin version: Itō

You should now have this in your bibliographic record:
100    _aItō
       _92646

4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:

100    _7ba0yda0y
       _a伊東豊雄
       _92646

The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.

5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'

6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:

100    _7ba0yba0y
       _aItō
       _92646

7) Edit the authority again, but set the $7 of the latin field to its short
form:

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba
       _aItō

8) Check that the bibliographic record still has the latin field reported:

100    _7ba
       _aItō
       _92646

9) prove koha/t/db_dependent/Authority/Merge.t

10) Sign-off :)

Sponsored-by: ENSA
Comment 2 Frédéric Demians 2024-10-11 15:00:34 UTC
Very interesting! Two remarks/demands:

- Could you develop the syspref description? Something with sample, and explaining the 2 language code forms, short and long.

- Wouldn't it make sense to have a repeatable languages selection? Let say I select 'ba,fa'. It means that I want first 'ba' authority, if not available then 'fa' authority, otherwise the 1st one.
Comment 3 Matthias Meusburger 2024-10-16 07:25:45 UTC
Created attachment 172802 [details] [review]
Bug 38142: Choose language to report from authority to bibliographic record.

Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.

However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.

French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d

This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.

When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.

If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.

The $7 in the authority field can be in a short or long form:

 - the short form is 2 characters long. Example: ba
 - the long form is 8 characters long, with the short form in position 5 and 6
   (starting from 1). Example: ba0yba0y

Test plan:
----------

1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable

2) Add an authority with several languages, like this (the order is important):

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba0yba0y
       _aItō

3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
 search for the authority, then Search main heading ($a only): contains Itō)

Note that you can choose which version you want when searching for the
authority, under the "Get it!" column

For this example, choose the latin version: Itō

You should now have this in your bibliographic record:
100    _aItō
       _92646

4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:

100    _7ba0yda0y
       _a伊東豊雄
       _92646

The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.

5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'

6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:

100    _7ba0yba0y
       _aItō
       _92646

7) Edit the authority again, but set the $7 of the latin field to its short
form:

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba
       _aItō

8) Check that the bibliographic record still has the latin field reported:

100    _7ba
       _aItō
       _92646

9) prove koha/t/db_dependent/Authority/Merge.t

10) Sign-off :)

Sponsored-by: ENSA
Comment 4 Matthias Meusburger 2024-10-16 07:28:22 UTC
(In reply to Frédéric Demians from comment #2)
> Very interesting! Two remarks/demands:
> 
> - Could you develop the syspref description? Something with sample, and
> explaining the 2 language code forms, short and long.

Agreed. The patch has been updated with a more descriptive syspref.

> 
> - Wouldn't it make sense to have a repeatable languages selection? Let say I
> select 'ba,fa'. It means that I want first 'ba' authority, if not available
> then 'fa' authority, otherwise the 1st one.

This is an interesting idea, but maybe perfect is the enemy of good :)
I this this could be added later in another BZ if needed.
Comment 5 Frédéric Demians 2024-10-17 17:36:41 UTC
Created attachment 172923 [details] [review]
Bug 38142: Choose language to report from authority to bibliographic record.

Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.

However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.

French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d

This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.

When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.

If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.

The $7 in the authority field can be in a short or long form:

 - the short form is 2 characters long. Example: ba
 - the long form is 8 characters long, with the short form in position 5 and 6
   (starting from 1). Example: ba0yba0y

Test plan:
----------

1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable

2) Add an authority with several languages, like this (the order is important):

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba0yba0y
       _aItō

3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
 search for the authority, then Search main heading ($a only): contains Itō)

Note that you can choose which version you want when searching for the
authority, under the "Get it!" column

For this example, choose the latin version: Itō

You should now have this in your bibliographic record:
100    _aItō
       _92646

4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:

100    _7ba0yda0y
       _a伊東豊雄
       _92646

The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.

5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'

6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:

100    _7ba0yba0y
       _aItō
       _92646

7) Edit the authority again, but set the $7 of the latin field to its short
form:

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba
       _aItō

8) Check that the bibliographic record still has the latin field reported:

100    _7ba
       _aItō
       _92646

9) prove koha/t/db_dependent/Authority/Merge.t

10) Sign-off :)

Sponsored-by: ENSA

Works smoothly, both with UNIMARC/MARC21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Comment 6 Lucas Gass (lukeg) 2025-01-31 20:12:00 UTC
Doesn't apply cleanly to main
Comment 7 Matthias Meusburger 2025-02-04 14:42:51 UTC
Created attachment 177508 [details] [review]
Bug 38142: Choose language to report from authority to bibliographic record.

Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.

However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.

French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d

This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.

When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.

If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.

The $7 in the authority field can be in a short or long form:

 - the short form is 2 characters long. Example: ba
 - the long form is 8 characters long, with the short form in position 5 and 6
   (starting from 1). Example: ba0yba0y

Test plan:
----------

1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable

2) Add an authority with several languages, like this (the order is important):

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba0yba0y
       _aItō

3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
 search for the authority, then Search main heading ($a only): contains Itō)

Note that you can choose which version you want when searching for the
authority, under the "Get it!" column

For this example, choose the latin version: Itō

You should now have this in your bibliographic record:
100    _aItō
       _92646

4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:

100    _7ba0yda0y
       _a伊東豊雄
       _92646

The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.

5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'

6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:

100    _7ba0yba0y
       _aItō
       _92646

7) Edit the authority again, but set the $7 of the latin field to its short
form:

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba
       _aItō

8) Check that the bibliographic record still has the latin field reported:

100    _7ba
       _aItō
       _92646

9) prove koha/t/db_dependent/Authority/Merge.t

10) Sign-off :)

Sponsored-by: ENSA

Works smoothly, both with UNIMARC/MARC21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Comment 8 Matthias Meusburger 2025-02-04 14:43:32 UTC
Thanks Lucas, the patch has been rebased on main.
Comment 9 Matthias Meusburger 2025-04-01 12:58:32 UTC
Created attachment 180178 [details] [review]
Bug 38142: Choose language to report from authority to bibliographic record.

Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.

However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.

French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d

This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.

When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.

If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.

The $7 in the authority field can be in a short or long form:

 - the short form is 2 characters long. Example: ba
 - the long form is 8 characters long, with the short form in position 5 and 6
   (starting from 1). Example: ba0yba0y

Test plan:
----------

1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable

2) Add an authority with several languages, like this (the order is important):

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba0yba0y
       _aItō

3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
 search for the authority, then Search main heading ($a only): contains Itō)

Note that you can choose which version you want when searching for the
authority, under the "Get it!" column

For this example, choose the latin version: Itō

You should now have this in your bibliographic record:
100    _aItō
       _92646

4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:

100    _7ba0yda0y
       _a伊東豊雄
       _92646

The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.

5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'

6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:

100    _7ba0yba0y
       _aItō
       _92646

7) Edit the authority again, but set the $7 of the latin field to its short
form:

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba
       _aItō

8) Check that the bibliographic record still has the latin field reported:

100    _7ba
       _aItō
       _92646

9) prove koha/t/db_dependent/Authority/Merge.t

10) Sign-off :)

Sponsored-by: ENSA

Works smoothly, both with UNIMARC/MARC21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Comment 10 Martin Renvoize (ashimema) 2025-04-01 16:24:19 UTC
Created attachment 180222 [details] [review]
Bug 38142: Choose language to report from authority to bibliographic record.

Currently, when reporting authority field to the bibliographic records,
the merge function uses the first matching authority field.

However, authorities can provide multiple versions of the same field, with
different languages, identified by a code in the $7 field.

French reference for languages codes for example, but any code system could be
used:
https://documentation.abes.fr/sudoc/formats/unmb/DonneesCodees/CodesZone104.htm#$d

This patch introduces the LanguageToReportOnMerge system preference, which
allows to specify a language code.

When set, if there is an authority field with the given language code, it will
be used to report to the bibliographic record.

If the system preference is not set, or there is no matching field, the default
behavior will apply: the first field will be used.

The $7 in the authority field can be in a short or long form:

 - the short form is 2 characters long. Example: ba
 - the long form is 8 characters long, with the short form in position 5 and 6
   (starting from 1). Example: ba0yba0y

Test plan:
----------

1) Edit the 'Personal Name' authority type framework to make the 100 field
repeatable

2) Add an authority with several languages, like this (the order is important):

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba0yba0y
       _aItō

3) Add a bibliographic record linked to that authority.
(Click on the "tag editor" icon next to the 100$a field in the bibliographic record to
 search for the authority, then Search main heading ($a only): contains Itō)

Note that you can choose which version you want when searching for the
authority, under the "Get it!" column

For this example, choose the latin version: Itō

You should now have this in your bibliographic record:
100    _aItō
       _92646

4) Edit the authority, you don't have to change anything, just edit and save.
Note that your bibliographic record now looks like this:

100    _7ba0yda0y
       _a伊東豊雄
       _92646

The first 100 field of the authority has been reported back to the bibliographic
record, regardless of what you chose when selecting the authority.

5) Apply the patch, set the LanguageToReportOnMerge system preference to: 'ba'

6) Edit the authority again, and check that your bibliographic record now looks
like this, according to the system preference:

100    _7ba0yba0y
       _aItō
       _92646

7) Edit the authority again, but set the $7 of the latin field to its short
form:

100    _7ba0yda0y
       _a伊東豊雄
100    _7ba
       _aItō

8) Check that the bibliographic record still has the latin field reported:

100    _7ba
       _aItō
       _92646

9) prove koha/t/db_dependent/Authority/Merge.t

10) Sign-off :)

Sponsored-by: ENSA

Works smoothly, both with UNIMARC/MARC21
Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 11 Katrin Fischer 2025-04-24 18:25:45 UTC
This might work for MARC21, but I don't think it should:

In MARC21 the 1xx fields are all marked as non-repeatable. And Koha's frameworks for MARC21 reflect that:
https://www.loc.gov/marc/authority/ad1xx3xx.html

And the $7 is used otherwise: $7 - Data provenance (R)

As we usually expect the 1xx to not be repeated I am not sure what kind of problems could arise if it is.

It looks like in UNIMARC authority this is OK:
https://repository.ifla.org/server/api/core/bitstreams/3a3df76b-dad8-4a41-95ee-5198813784fa/content

1) I feel we should  mark this as a UNIMARC feature in the system preference text. (please argue strongly or follow-up)

2) I am not sure if "reported" is clear here as the verb in both the system preference name and description. It's more like it will be the one that is used/copied to the bibliographic record if I understand correctly? (question to native speakers). (not a blocker, but might confuse translators?)

3) Please use say_success in your database updates... fixed before pushing (done)
Comment 12 Katrin Fischer 2025-04-25 06:48:49 UTC
Pushed for 25.05!

Well done everyone, thank you!
Comment 13 paxed 2025-05-06 11:10:50 UTC
What is this $7 format? I don't see anything like it described on https://www.loc.gov/marc/authority/adapndxh.html
Comment 14 Katrin Fischer 2025-05-06 11:46:24 UTC
I don't think it exists in MARC21, see my previous comment. That's why I was asking to mark this as a UNIMARC only feature (no follow-up from developer yet...)
Comment 15 paxed 2025-05-07 05:31:06 UTC
I have to agree with the "reported" sounding weird. Perhaps it should be something like "used to select the authority record language"
Comment 16 Katrin Fischer 2025-05-07 07:03:34 UTC
I nudged Matts on Mattermost.

I think to be more precise it would be something like:

... the language code of the main heading to use
Comment 17 Matthias Meusburger 2025-05-20 10:05:43 UTC
Created attachment 182652 [details] [review]
Bug 38142, Follow-up: Choose language to report from authority to bibliographic record.

 - Rename LanguageToReportOnMerge system preference to LanguageToUseOnMerge
 - Use "use" instead of "report" in system preference description
 - Mark this feature as UNIMARC-only.
Comment 18 Katrin Fischer 2025-05-20 12:51:01 UTC
Matts, can you please double check? Your follow-up patch (only applied the second) doesn't apply cleanly on main. Also the commit title will fail QA tools ;)
Comment 19 Matthias Meusburger 2025-05-20 13:02:42 UTC
Created attachment 182671 [details] [review]
Bug 38142, Follow-up: Choose language to report from authority to bibliographic record.

- Rename LanguageToReportOnMerge system preference to LanguageToUseOnMerge
 - Use "use" instead of "report" in system preference description
 - Mark this feature as UNIMARC-only.
Comment 20 Katrin Fischer 2025-05-20 13:15:24 UTC
Fixed the commit title (Bug 3814, ;)) and added a follow-up for sysprefs.sql.