Bug 37196 - Automatically add punctuation to Marc21 title fields
Summary: Automatically add punctuation to Marc21 title fields
Status: ASSIGNED
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: unspecified
Hardware: All All
: P4 enhancement
Assignee: Alexander Wagner
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 21303
  Show dependency treegraph
 
Reported: 2024-06-26 12:12 UTC by Alexander Wagner
Modified: 2024-07-18 14:47 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 37196: Add automatic punctuation for Marc-245 (23.21 KB, patch)
2024-07-05 12:50 UTC, Alexander Wagner
Details | Diff | Splinter Review
Bug 37196: Add automatic punctuation for Marc title fields (8.00 KB, patch)
2024-07-18 14:47 UTC, Alexander Wagner
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Wagner 2024-06-26 12:12:51 UTC
If the record leader 18 specifies `c` or `n` the Marc record in question does not hold any punctuation chars. IOW the record would look like

```
245 10 |a Feynman-Kac-type theorems and Gibbs measures on path space
       |b with applications to rigorous quantum field theory
       |c József Lörinczi ; Fumio Hiroshima ; Volker Betz
```

(Sample from DNB: http://d-nb.info/988967189/about/marcxml)

Currently, Koha will display this as

```
Feynman-Kac-type theorems and Gibbs measures on path space with applications to rigorous quantum field theory József Lörinczi ; Fumio Hiroshima ; Volker Betz
```

while it should actually be displayed like

```
Feynman-Kac-type theorems and Gibbs measures on path space : with applications to rigorous quantum field theory / József Lörinczi ; Fumio Hiroshima ; Volker Betz
```

In this example, if leader 18 == `c`|`n`, Koha should

- add the missing ` : ` between title and remainder of title
- Add ` / ` before the statement of responsibility
- Add a `<span class="title_remainder">` around the remainder of title to allow for formatting

Analogously, it should treat the other fields of 245.

Use https://www.loc.gov/aba/pcc/documents/isbdmarc2016.pdf as a guideline.
Comment 1 Esther Melander 2024-06-27 15:35:07 UTC
I can see how this might improve the quality of MARC records. How would this impact the ability of catalogers to edit the 245 and have the edits stick? Perhaps a cataloger does not want those kinds of edits? Is this something that could be disabled in system preferences.
Comment 2 Katrin Fischer 2024-06-27 15:38:54 UTC
Hi Esther, this wouldn't affect cataloguing. The data would not be changed, the punctuation only added for display.
Comment 3 Alexander Wagner 2024-07-01 07:34:39 UTC
Hi Esther!

I'd not change anything on the cataloguing end, but only on the _display_ side of the record after cataloguing.

Furthermore, as it respects the Marc leader field 18 it is up to the cataloguer to decide if she add punctuation or not. She should just set the leader properly.

If you decide to add all punctuations in your record you'd set the leader to the proper value of `a` or `i` and _nothing_ will change.

If you decide not to set the punctuation you set the leader accordingly to `c` or `n` and Koha should add the punctuation (as best as it can) automatically.

This even allows for a mix of records in your catalogue. Records that hold punctuation are displayed with the punctuation _as is_, records without get it added. So, if you notice that your record in question is a corner case where it is not possible to add the punctuation properly by the machine you could still add it by hand.

Please mind also in this minute ticket here I'll only touch 245. So it's only a tiny part of the longer standing ticket 21303.
Comment 4 Alexander Wagner 2024-07-05 07:22:29 UTC
There are at least two XLS-files that need to be adopted:

- `MARC21slim2OPACDetail.xsl` for the details page
- `MARC21slim2OPACResults.xsl` for the results list

With some minor refactoring both can share the same code for field 245, once its rendering is moved into a separate template. This would have the nice side effect to ease maintenance.
Comment 5 Alexander Wagner 2024-07-05 09:28:03 UTC
There are some cases where a proper punctuation can not be added automatically. E.g. one of the ktd-examples is

```
245 10
       |a Dàin do Eimhir =
       |b Poems to Eimhir /
       |c Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte.
```

where subfield |b gives a translation of the main title in |a that should be separated by `=` rather than `:`. As the machine can not distinguish this from a subtitle it will be required to add the proper punctuation and set the leader 18 accordingly to `a`, `i` or `u` which will switch _off_ the automatic addition of punctuation.
Comment 6 Alexander Wagner 2024-07-05 12:50:30 UTC
Created attachment 168536 [details] [review]
Bug 37196: Add automatic punctuation for Marc-245

If the Marc Leader 18 is set to `c` or `n` add the necessary punctuation
for the Marc21 field 245 automatically. Otherwise assume the punctuation
exists and use the subfield data _as is_. The following subfields are
handled: bcfghknps

Additionally, all title parts are enclosed in individual `<span>` tags
with proper class assignments to allow for easy styling of all the
individual parts.

Note: some special cases (like title translations that should get an `=`
instead of a `:`) can not be handled fully automatically.

The change applies to:
- OPAC results
- OPAC details
- intranet results
- intranet details

Testplan:

1. Start `ktd`
2. Check a record with additional title fields.
   Examples:
   - biblionumber 55:
     Dàin do Eimhir = Poems to Eimhir / Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte.
   - biblionumber 58
     'More fruitful than the soil' : army, empire and the Scottish Highlands, 1715-1815 / Andrew Mackillop.
   Those should display correctly, as they hold all punctuations _in_
   the record. (Marc Leader 18 set to `a`).
3. Apply the patch
4. Call up the records again
   They should display unchanged as the Leader 18 is still set to `a`
   => no automatic punctuation is applied
5. Edit both records and only set the Leader 18 to either `c` or `n`.
   (Note: this is _incorrect_ wrt the record semantic.)

   All punctuation chars appear _twice_ as Koha adds them automatically
   as instructed even though they are there. The title display like

   - Dàin do Eimhir = : Poems to Eimhir / / Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte.
   - 'More fruitful than the soil' : : army, empire and the Scottish Highlands, 1715-1815 / / Andrew Mackillop.

   Note that for biblionumber 55 Koha displays `= :` as it is not
   possible to know automatically that there should be an `=` instad of
   the `:`.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library
Comment 7 Heather 2024-07-16 15:16:57 UTC
Hi!

I am late to the party, but in the case of a record with a 245 that contains a parallel title, e.g.:

> 245 10
>        |a Dàin do Eimhir =
>        |b Poems to Eimhir /
>        |c Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher
> Whyte.
> ```
> 
> where subfield |b gives a translation of the main title in |a that should be
> separated by `=` rather than `:`. As the machine can not distinguish this
> from a subtitle...

The record should contain an 041 field with a first indicator 1, "Item is or includes a translation," https://www.loc.gov/marc/bibliographic/bd041.html

Just mentioning this in case it's useful for the machine to use this to determine if the 245 $b is a parallel title (so add the equals sign) or a subtitle (so add the colon).

And, BTW, I think this enhancement is wonderful!!!  Signed,
Real Cataloger h2
Comment 8 Heather 2024-07-16 16:19:04 UTC
Oh!  And I forgot to mention that the 041 would also have more than one $a to indicate that it has more than one language in it that are translations.

(And there would be rare examples where this isn't the case.)

--h2
Comment 9 Alexander Wagner 2024-07-17 09:00:15 UTC
> The record should contain an 041 field with a first indicator 1, "Item is or includes a translation,"

I just checked the sample record form Koha / ktd wrt 041 and the example (biblionumber 55)

```
245 10 |a Dàin do Eimhir =
       |b Poems to Eimhir /
       |c Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte.
```

that displays like

Dàin do Eimhir = Poems to Eimhir / Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte. 

and find the following usage:

```
041 0_ |a gla
       |a eng
       |h gla
```

and no `041 1_`.

There is a second example for the title (biblionumber 57) that displays as

Dàin do Eimhir / Somhairle MacGill-Eain = Poems to Eimhir / Sorley MacLean ; edited by Christopher Whyte. 

```
041 0_ |a gleeng
245 10 |a Dàin do Eimhir /
       |c Somhairle MacGill-Eain =
       |b Poems to Eimhir / Sorley MacLean ; edited by Christopher Whyte.
```

(For me the usage of `041` in this example looks _interesting_.)

I am unsure if it is correct to conclude, that if `041 ??` lists _more than one language_ one should assume that `245` holds a translated title in `|b`.

Side note: the proposed change would allow for the usage of 245 fields _as is_ provided the leader specifies that punctuation _is_ included.
Comment 10 Katrin Fischer 2024-07-17 09:07:39 UTC
It looks a bit like there were accidentally 2 codes added to one $a. gle and eng (it's 3 letter codes always). Maybe should have been a repeated one instead.

Data is often not as clean we we'd like or not as standard. I am not sure if we should invest a lot in the parallel title case here. 

I can say that for our union catalog (K10plus) that the = is included in the data although the punctuation is omitted in general. Maybe because it's not possible to guess it reliably?
Comment 11 Alexander Wagner 2024-07-17 09:13:50 UTC
> It looks a bit like there were accidentally 2 codes added to one $a. gle and eng (it's 3 letter codes always). Maybe should have been a repeated one instead.

Agree.

I usually treat this as `error` in my record migration routines and split them accordingly. As it's an official Koha example record however, I wonder if there is some area on this planet where it's done like that. (I know that we had quite a few of these guys in our old Aleph-records.)

> Data is often not as clean we we'd like or not as standard.

Yepp.

> I can say that for our union catalog (K10plus) that the = is included in the data although the punctuation is omitted in general. Maybe because it's not possible to guess it reliably?

I think so. Also note that both records at least (for me) seem to specify the same title but the translation of the title and the responsibility statement are handled differently. Easy enough to understand for humans, hard for machines.
Comment 12 Alexander Wagner 2024-07-18 14:47:53 UTC
Created attachment 169139 [details] [review]
Bug 37196: Add automatic punctuation for Marc title fields

Add a new MARC filter to automatically inject punctuation for MARC
fields 245, 246 and 246 if the MARC Leader 18 is set to `c` or `n`.
Otherwise assume the punctuation exists and use the subfield data _as
is_. Hook in this filter before any XSL tranformations happen, so all
subsequent processing can assume that punctuation _exists_.

The following subfields are handled:

- 245: bcdefghknps
- 246/247: bfghinqrt

Note: some special cases (like title translations that should get an `=`
instead of a `:`) can not be handled fully automatically.

Testplan:

1. Start `ktd`
2. Check a record with additional title fields.
   Examples:
   - biblionumber 55:
     DÃ in do Eimhir = Poems to Eimhir / Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte.
   - biblionumber 58
     'More fruitful than the soil' : army, empire and the Scottish Highlands, 1715-1815 / Andrew Mackillop.
   Those should display correctly, as they hold all punctuations _in_
   the record. (Marc Leader 18 set to `a`).
3. Apply the patch
4. Call up the records again
   They should display unchanged as the Leader 18 is still set to `a`
   => no automatic punctuation is applied
5. Edit both records and only set the Leader 18 to either `c` or `n`.
   (Note: this is _incorrect_ wrt the record semantic.)

   All punctuation chars appear _twice_ as Koha adds them automatically
   as instructed even though they are there. The title display like

   - DÃ in do Eimhir = : Poems to Eimhir / / Somhairle MacGill-Eain = Sorley MacLean ; edited by Christopher Whyte.
   - 'More fruitful than the soil' : : army, empire and the Scottish Highlands, 1715-1815 / / Andrew Mackillop.

   Note that for biblionumber 55 Koha displays `= :` as it is not
   possible to know automatically that there should be an `=` instad of
   the `:`.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library