Bug 37325 - Automatically add punctuation to Marc21 field 260 / 264
Summary: Automatically add punctuation to Marc21 field 260 / 264
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: MARC Bibliographic data support (show other bugs)
Version: unspecified
Hardware: All All
: P4 enhancement
Assignee: Alexander Wagner
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 21303
  Show dependency treegraph
 
Reported: 2024-07-11 11:42 UTC by Alexander Wagner
Modified: 2024-11-13 13:10 UTC (History)
1 user (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 37325: Add punctuation for 260 (7.64 KB, patch)
2024-11-11 13:41 UTC, Alexander Wagner
Details | Diff | Splinter Review
Bug 37325: fix XSLT for automatic punctuation of 260 (21.42 KB, patch)
2024-11-11 13:41 UTC, Alexander Wagner
Details | Diff | Splinter Review
Bug 37325: Add punctuation for 260 and 264 (8.12 KB, patch)
2024-11-13 12:12 UTC, Alexander Wagner
Details | Diff | Splinter Review
Bug 37325: fix XSLT for automatic punctuation of 260 and 264 (31.69 KB, patch)
2024-11-13 12:12 UTC, Alexander Wagner
Details | Diff | Splinter Review
Bug 37325: Add punctuation for 260 and 264 (8.12 KB, patch)
2024-11-13 13:10 UTC, Alexander Wagner
Details | Diff | Splinter Review
Bug 37325: fix XSLT for automatic punctuation of 260 and 264 (31.69 KB, patch)
2024-11-13 13:10 UTC, Alexander Wagner
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Wagner 2024-07-11 11:42:52 UTC
If the record leader 18 specifies `c` or `n` the Marc record in question does not hold any punctuation chars. IOW the record would look like

```
264 _1 |a Berlin
       |a Boston, Mass.
       |b de Gruyter
       |c 2011
```

This should display like

Berin ; Boston, Mass. : de Gruyter, 2011

A similar handling is required for both 260 and 264.
Comment 1 Alexander Wagner 2024-07-12 10:19:24 UTC
More samples:

```
260 __ |a Paris :
       |b Impr. Vincent,$c1798
       |a [i.e. Bruxelles :
       |b Moens,
       |c **1883]								

260 __ |a Bethesda, Md. :
       |b Toxicology Information Program, National Library of Medicine [producer] ;
       |a Springfield, Va. :
       |b National Technical Information Service [distributor],
       |c 1974-$e(Oak Ridge, Tenn. :
       |f **Oak Ridge National Laboratory [generator])
```
Comment 2 Alexander Wagner 2024-07-12 14:48:19 UTC
Start exploring Filters for this

https://git.koha-community.org/Koha-community/Koha/src/branch/main/Koha/Filter/MARC

A filter would add the punctuation _before_ the fields are passed on to the XSLT, so the final step would always see a (hopefully) properly punctuated record.

This should make it easier to handle cases where the subfields effectively form groups of logical entries like `Paris : Impr. Vincent, 1798`.
Comment 3 Alexander Wagner 2024-10-29 16:45:14 UTC
As fields from 260 are used to build links into Koha adding the punctuation actually clutters the search terms with these chars. IOW the actual search triggered currently is not for `Paris` but for `Paris : `.

This is also true for all records that hold the punctuation, so adding it automatically does not degrade the situation and Koha seems to cope with that quite well. (This issue seems to have been discussed in https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=1807 some time ago already and the current state seems _good enough_.)

To improve on this end one would need to add proper functionality to the XSLT to strip (the just added or stored) punctuation. Being limited to XSLT 1 a complete solution might get a bit complex (no regexps...)
Comment 4 Alexander Wagner 2024-11-11 13:41:48 UTC
Created attachment 174325 [details] [review]
Bug 37325: Add punctuation for 260

Add punctuation automatically for tag 260 (Imprint), if the record
leader defines that punctuation is _omitted_ from the fields. The filter
does not act on records that hold punctuation and say so in leader 18.

- Hook up `Puncutation.pm` with `XSLT.pm`
- Add rules for automatic punctuation of Marc21 field 260
  according to
  https://www.loc.gov/aba/pcc/documents/isbdmarc2016.pdf
- Subfields handled: a, b, c, f, g, r, t, q

Test plan:

1. Select a record (A) with a properly filled and punctuated 260 field
2. Check that the punctuation display correctly in intranet and OPAC
   interface
3. Select a record (B) without punctuation OR remove the punctuation
   from the subfields of field 260. Check that leader 18 is set to `c`
   or `n`, IOW that this is a semantically correct record.
4. Check that _no_ punctuation chars get displayed in OPAC or intranet
5. Apply the patch

3. Edit the record (A)
4. Check that the Marc leader position 18 is _not_ set to `c` or `n`
5. Set Marc leader 18 to `c` or `n`
   (`c`: ISBD punctuation omitted / `n` Non-ISBD punctuation omitted)
4. Save the record
5. Check that all punctuation chars appear twice in intranet as well as
   OPAC interface
6. Check record (B) in intranet  and OPAC interface. It should now show
   automatically added, proper punctuation.

Note: there may be complex cases where it is not possible to properly
add punctuation automatically from the subfield semantics alone. In
these cases it is still possible to add punctuation to the record and
specify this in leader 18. Then the filter will not change the record at
all.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library
Comment 5 Alexander Wagner 2024-11-11 13:41:51 UTC
Created attachment 174326 [details] [review]
Bug 37325: fix XSLT for automatic punctuation of 260

XSLT uses subfields for display _as they are_. Sometimes, these values
are used to build queries against Koha based on the field content. These
queries may hold punctuation chars that could potentially confuse the
search.

- Add simple rules to strip punctuation chars in the search terms for
  OPAC and intranet interfaces.

Test plan:

1. Call up any record holding a 260 field in Koha
2. Check the URLs of queries generated from the subfields of 260. These
   show search terms including punctuation chars (`,:` etc., e.g.
   `Edinburg :`)
3. Apply the patch
4. Punctuation chars should vanish from the URL and only the search term
   should survive (e.g. `Edinburg`)

This fix acts on _all_ records and is not restricted to records that got
their punctuation automatically.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library
Comment 6 Alexander Wagner 2024-11-13 12:12:32 UTC
Created attachment 174467 [details] [review]
Bug 37325: Add punctuation for 260 and 264

Add punctuation automatically for tag 260 (Imprint), if the record
leader defines that punctuation is _omitted_ from the fields. The filter
does not act on records that hold punctuation and say so in leader 18.

- Hook up `Puncutation.pm` with `XSLT.pm`
- Add rules for automatic punctuation of Marc21 field 260
  according to
  https://www.loc.gov/aba/pcc/documents/isbdmarc2016.pdf
- Subfields handled: a, b, c, f, g, r, t, q

Note: there may be complex cases where it is not possible to properly
add punctuation automatically from the subfield semantics alone. In
these cases it is still possible to add punctuation to the record and
specify this in leader 18. Then the filter will not change the record at
all.

Test plan:

1. Select a record (A) with a properly filled and punctuated 260 field
2. Check that the punctuation display correctly in intranet and OPAC
   interface
3. Select a record (B) without punctuation OR remove the punctuation
   from the subfields of field 260. Check that leader 18 is set to `c`
   or `n`, IOW that this is a semantically correct record.
4. Check that _no_ punctuation chars get displayed in OPAC or intranet
5. Apply the patch

6. Edit the record (A)
7. Check that the Marc leader position 18 is _not_ set to `c` or `n`
8. Set Marc leader 18 to `c` or `n`
   (`c`: ISBD punctuation omitted / `n` Non-ISBD punctuation omitted)
9. Save the record
10. Check that all punctuation chars appear twice in intranet as well as
   OPAC interface
11. Check record (B) in intranet  and OPAC interface. It should now show
   automatically added, proper punctuation.

For field 264: follow the procedure just outlined but use field 264
instead. If 264 is present, Koha will prefer it over 260.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library
Comment 7 Alexander Wagner 2024-11-13 12:12:34 UTC
Created attachment 174468 [details] [review]
Bug 37325: fix XSLT for automatic punctuation of 260 and 264

XSLT uses subfields for display _as they are_. Sometimes, these values
are used to build queries against Koha based on the field content. These
queries may hold punctuation chars that could potentially confuse the
search.

- Add simple rules to strip punctuation chars in the search terms for
  OPAC and intranet interfaces.

Test plan:

1. Call up any record holding a 260 field in Koha
2. Check the URLs of queries generated from the subfields of 260. These
   show search terms including punctuation chars (`,:` etc., e.g.
   `Edinburg :`)
3. Apply the patch
4. Punctuation chars should vanish from the URL and only the search term
   should survive (e.g. `Edinburg`)

For field 264 follow the procedures just outlined.

This fix acts on _all_ records and is not restricted to records that got
their punctuation automatically.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library
Comment 8 Alexander Wagner 2024-11-13 13:10:54 UTC
Created attachment 174470 [details] [review]
Bug 37325: Add punctuation for 260 and 264

Add punctuation automatically for tag 260 (Imprint), if the record
leader defines that punctuation is _omitted_ from the fields. The filter
does not act on records that hold punctuation and say so in leader 18.

- Hook up `Puncutation.pm` with `XSLT.pm`
- Add rules for automatic punctuation of Marc21 field 260
  according to
  https://www.loc.gov/aba/pcc/documents/isbdmarc2016.pdf
- Subfields handled: a, b, c, f, g, r, t, q

Note: there may be complex cases where it is not possible to properly
add punctuation automatically from the subfield semantics alone. In
these cases it is still possible to add punctuation to the record and
specify this in leader 18. Then the filter will not change the record at
all.

Test plan:

1. Select a record (A) with a properly filled and punctuated 260 field
2. Check that the punctuation display correctly in intranet and OPAC
   interface
3. Select a record (B) without punctuation OR remove the punctuation
   from the subfields of field 260. Check that leader 18 is set to `c`
   or `n`, IOW that this is a semantically correct record.
4. Check that _no_ punctuation chars get displayed in OPAC or intranet
5. Apply the patch

6. Edit the record (A)
7. Check that the Marc leader position 18 is _not_ set to `c` or `n`
8. Set Marc leader 18 to `c` or `n`
   (`c`: ISBD punctuation omitted / `n` Non-ISBD punctuation omitted)
9. Save the record
10. Check that all punctuation chars appear twice in intranet as well as
   OPAC interface
11. Check record (B) in intranet  and OPAC interface. It should now show
   automatically added, proper punctuation.

For field 264: follow the procedure just outlined but use field 264
instead. If 264 is present, Koha will prefer it over 260.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library
Comment 9 Alexander Wagner 2024-11-13 13:10:56 UTC
Created attachment 174471 [details] [review]
Bug 37325: fix XSLT for automatic punctuation of 260 and 264

XSLT uses subfields for display _as they are_. Sometimes, these values
are used to build queries against Koha based on the field content. These
queries may hold punctuation chars that could potentially confuse the
search.

- Add simple rules to strip punctuation chars in the search terms for
  OPAC and intranet interfaces.

Test plan:

1. Call up any record holding a 260 field in Koha
2. Check the URLs of queries generated from the subfields of 260. These
   show search terms including punctuation chars (`,:` etc., e.g.
   `Edinburg :`)
3. Apply the patch
4. Punctuation chars should vanish from the URL and only the search term
   should survive (e.g. `Edinburg`)

For field 264 follow the procedures just outlined.

This fix acts on _all_ records and is not restricted to records that got
their punctuation automatically.

Sponsored-by: Deutsches Elektronen-Synchrotron DESY, Library