Bug 40300 - Multiple semi-colons in series part name (830$p)
Summary: Multiple semi-colons in series part name (830$p)
Status: In Discussion
Alias: None
Product: Koha
Classification: Unclassified
Component: OPAC (show other bugs)
Version: Main
Hardware: All All
: P5 - low minor
Assignee: Bernard
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-07-03 10:28 UTC by Bernard
Modified: 2025-08-07 13:12 UTC (History)
5 users (show)

See Also:
GIT URL:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This fixes the display of the 830 series tag in the staff interface and OPAC. An 830 series tag which has a part $p subfield displays multiple semicolons if the subfield already contains one. Example: - A record with an 830 tag with these subfields: $a Analecta Gregoriana, Series Facultatis Philosophiae. $p Sectio B ; $v n. 14. - Before this patch is shows as: Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; ; n. 14. - After the patch is shows as: Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; n. 14.
Version(s) released in:
Circulation function:


Attachments
Bug 40300: Add chop punctuation to PartName () in Part xslt templates (2.06 KB, patch)
2025-07-03 10:34 UTC, Bernard
Details | Diff | Splinter Review
Bug 40300: Add chop punctuation to PartName () in Part xslt templates (2.66 KB, patch)
2025-07-07 11:28 UTC, Bernard
Details | Diff | Splinter Review
Bug 40300: Add chop punctuation to PartName () in Part xslt templates (2.70 KB, patch)
2025-07-07 18:44 UTC, David Nind
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Bernard 2025-07-03 10:28:22 UTC
In both intranet and OPAC full detail display an 830 series field which has a part $p displays multiple semi-colons if the field contains one. There is a part template which needs to use chopPunctuation.

1. Add series field 830 with 

$a Analecta Gregoriana, Series Facultatis Philosophiae.
$p Sectio B ;
$v n. 14.

2. Observe OPAC or intranet detail record:

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; ; n. 14.

3. Should be 

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; n. 14.
Comment 1 Bernard 2025-07-03 10:34:26 UTC
Created attachment 183742 [details] [review]
Bug 40300: Add chop punctuation to PartName () in Part xslt templates
Comment 2 Bernard 2025-07-03 10:36:55 UTC
1. Add series field 830 with 

$a Analecta Gregoriana, Series Facultatis Philosophiae.
$p Sectio B ;
$v n. 14.

2. Observe OPAC or intranet detail record:

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; ; n. 14.

3. Apply patch and restart_all

4. Observe OPAC or intranet detail record again - should be

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; n. 14.
Comment 3 Alexander Wagner 2025-07-03 16:27:51 UTC
Probably a bit philosophical: While I am personally all for removing punctuation from records and adding them automatically, I do wonder if in your case Koha is not entering a strange mixture. IMHO

- if a punctuation _is there_ the logic should be to keep and use it.
- if no punctuation is there it should add it.

IOW while I see what your code does and I also perfectly agree that having ` ; ; ` is wrong _and_ ugly, I do wonder why the ` ; ` in `|p` is different from the `.` in `|a` or `|v` and thus treated differently.

So, I think instead of chopping the punctuation from `|p` and add it again automatically, the code should look at leader 18 and decide if it should do anything wrt the punctuation in the first place. And if the leader says "all chars are there" it should just do nothing and leave it as is. (My guess would be your records states that it holds those chars.)

BTW: I started working on automatic punctuation in https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21303. After playing around with XSLT and some discussion with Cait we came to the conclusion (which might be wrong) that it is probably _a lot_ easier to do this punctuation stuff by means of a Marc Filter in Perl especially as there are some complex things out there. Cf https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37196, https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37325 and their tests for examples and an implementation for a few fields.
Comment 4 David Nind 2025-07-04 21:16:06 UTC
Hi Bernard.

It would be great if your patch could include the description of the problem the bug addresses and the test plan (see the commit message guidelines https://wiki.koha-community.org/wiki/Commit_messages).

Let us know on the bug or in the Mattermost chat development channel if you need help with this.

David
Comment 5 David Nind 2025-07-04 21:17:35 UTC
Also added you as the assignee 8-)
Comment 6 Bernard 2025-07-07 11:28:56 UTC
Created attachment 183830 [details] [review]
Bug 40300: Add chop punctuation to PartName () in Part xslt templates

In both intranet and OPAC full detail display an 830 series field which has a part $p displays multiple semi-colons if the field contains one. There is a part template which needs to use chopPunctuation.
Test plan:
1. Add series field 830 with

$a Analecta Gregoriana, Series Facultatis Philosophiae.
$p Sectio B ;
$v n. 14.

2. Observe OPAC or intranet detail record:

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; ; n. 14.

3. Apply patch and restart_all

4. Observe OPAC or intranet detail record again - should be

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; n. 14.
Comment 7 Bernard 2025-07-07 11:34:37 UTC
(In reply to Alexander Wagner from comment #3)
> Probably a bit philosophical: While I am personally all for removing
> punctuation from records and adding them automatically, I do wonder if in
> your case Koha is not entering a strange mixture. IMHO
> 
> - if a punctuation _is there_ the logic should be to keep and use it.
> - if no punctuation is there it should add it.
> 
> IOW while I see what your code does and I also perfectly agree that having `
> ; ; ` is wrong _and_ ugly, I do wonder why the ` ; ` in `|p` is different
> from the `.` in `|a` or `|v` and thus treated differently.
> 
> So, I think instead of chopping the punctuation from `|p` and add it again
> automatically, the code should look at leader 18 and decide if it should do
> anything wrt the punctuation in the first place. And if the leader says "all
> chars are there" it should just do nothing and leave it as is. (My guess
> would be your records states that it holds those chars.)
> 
> BTW: I started working on automatic punctuation in
> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21303. After
> playing around with XSLT and some discussion with Cait we came to the
> conclusion (which might be wrong) that it is probably _a lot_ easier to do
> this punctuation stuff by means of a Marc Filter in Perl especially as there
> are some complex things out there. Cf
> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37196,
> https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=37325 and their
> tests for examples and an implementation for a few fields.

Interesting. I'm all for anything that can make this simpler and I like your idea of a separate punctuation.pm. I would be reluctant to have to rely on the LDR position 18 value as experience shows this may be set to "a" but that not be the reality. Perhaps a syspref making it an option?

I guess the issue is if we currently already use chopPunctuation at all in the xslt (we do), it should do it everywhere required. Or else not at all.  For now, pragmatically, I think I'd rather stick to fixing what we have.
Comment 8 David Nind 2025-07-07 18:44:45 UTC
Created attachment 183849 [details] [review]
Bug 40300: Add chop punctuation to PartName () in Part xslt templates

In both intranet and OPAC full detail display an 830 series field which has a part $p displays multiple semi-colons if the field contains one. There is a part template which needs to use chopPunctuation.
Test plan:
1. Add series field 830 with

$a Analecta Gregoriana, Series Facultatis Philosophiae.
$p Sectio B ;
$v n. 14.

2. Observe OPAC or intranet detail record:

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; ; n. 14.

3. Apply patch and restart_all

4. Observe OPAC or intranet detail record again - should be

Analecta Gregoriana, Series Facultatis Philosophiae. Sectio B ; n. 14.

Signed-off-by: David Nind <david@davidnind.com>
Comment 9 David Nind 2025-07-07 18:54:57 UTC
I've signed off.

However, I'm not knowledgeable enough about cataloging conventions and Koha to know what the "correct" way to do this is.

I know it has been discussed before, but the nuances are over my head 8-).

It may pay to ask in the Koha Mattermost cataloging channel to get some feedback from catalogers. 

Testing notes (using KTD):
1. Edit the BKS framework to make the 830 $a, $p, and $v subfields visible in the editor.
2. Edit a record in the sample data that uses the BKS framework, for example "Programming Perl".
Comment 10 cgresser 2025-07-11 10:54:21 UTC
I'm new to Koha, but have over 20 years of experience as a cataloguer. Anyone who doesn't know Koha but knows about Marc21 standards for cataloguing will first assume that a cataloguer of a Koha record of a series titles doesn't know their chops. However, as it is in _every_ series record which has an 830 field/tag, anyone will think that Koha is sub-standard. It would be great to avoid this impression, as Koha is _extremely_ versatile and also a great tool for professional cataloguing.

If this could be fixed, I think Christmas (Hanukah, Eid al-Fitr, or a choice of a festival which makes you happy) would come early.

Thank you for considering this for inclusion to the next version.
Comment 11 Alexander Wagner 2025-07-14 07:13:25 UTC
(In reply to Bernard from comment #7)
> (In reply to Alexander Wagner from comment #3)
> > Probably a bit philosophical: While I am personally all for removing
> > punctuation from records and adding them automatically, I do wonder if in
> > your case Koha is not entering a strange mixture. IMHO
> > 
> > - if a punctuation _is there_ the logic should be to keep and use it.
> > - if no punctuation is there it should add it.
> > 
[...]
> 
> Interesting. I'm all for anything that can make this simpler and I like your
> idea of a separate punctuation.pm. I would be reluctant to have to rely on
> the LDR position 18 value as experience shows this may be set to "a" but
> that not be the reality.

Hm, but we agree that the records you have in mind are "broken" wrt the idea of Marc in this regard. Shouldn't one fix the broken record then (IOW fix the leader)?

I think in general one will have a mixture of records in any given system over time, those that hold the punctuation and those that don't, right? They may just flood in during cataloguing if you ingest records from various sources. (AFAIK the SRU/Z39.50 importers do _not_ strip or add punctuation, so it depends on the source what you get.)

> Perhaps a syspref making it an option?

Uh. This adds another unrelated point to configure what to do with those funny chars and it would apply globally as well. One could head for a switch for 

- follow the leader
- ignore the leader and treat as "records have punctuation"
- ignore the leader and treat as "records don't have punctuation"

Where in the last case you'd chop everything "just in case".

This sounds a bit weird. 
 
> I guess the issue is if we currently already use chopPunctuation

I noticed that, indeed.

I was wondering all the time if this is not fixing a fix for a fix that might get another layer of plaster as soon as another issue pops up ;) That's the sole reason why I commented on this bug here that I noticed quite a few occurrences of chopping while some fields never see it.

> at all in the xslt (we do), it should do it everywhere required.

When I looked at Koha sample records for the test cases to `punctuation.pm` I found, that "strange things happen in cataloguing" that will not allow to add punctuation in _all_ corner cases automatically. Or actually the other way round: it is well possible and following the rules to construct a record that will not allow automatic punctuation. Hence, for those cases it might be necessary to add the punctuation on record level. But to give a proper display in those cases you'd have to leave the punctuation alone.

I think this is the reason behind the leader-switch.

> Or else not at all.

I tend to believe "not at  all" would be the way to go. Or at least if it's done in xslt it should follow the leader and if the display is wrong the offending leader can be fixed easily.
 
> For now, pragmatically, I think I'd rather stick to fixing what we have.

This is also ok with me if Koha thinks this is the way to go.

I just see some more issues down the winding road as chopping assumes you can _always_ add the punctuation automatically. (If so, why not strip it entirely on ingest and set the leader properly ;)
Comment 12 Marcel de Rooy 2025-08-01 08:39:10 UTC
Apart from discussing spaces or tabs, punctuation is a great alternative, haha :)

I did a git grep on xsl:text with some punct chars (discarding UNIMARC):

git grep "<xsl:text> [,;.:]" | grep -v UNIMARC
koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slimUtils.xsl:                <xsl:text> ; </xsl:text><xsl:value-of select="marc:subfield[@code='v']" />
koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slimUtils.xsl:                <xsl:text> ; </xsl:text><xsl:value-of select="marc:subfield[@code='v']" />
koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slimUtils.xsl:            <xsl:text> ; </xsl:text>
koha-tmpl/intranet-tmpl/prog/en/xslt/MARC21slimUtils.xsl:                <xsl:text> ; </xsl:text><xsl:value-of select="marc:subfield[@code='v']" />
koha-tmpl/opac-tmpl/bootstrap/en/xslt/MARC21slimUtils.xsl:                <xsl:text> ; </xsl:text><xsl:value-of select="marc:subfield[@code='v']" />
koha-tmpl/opac-tmpl/bootstrap/en/xslt/MARC21slimUtils.xsl:                <xsl:text> ; </xsl:text><xsl:value-of select="marc:subfield[@code='v']" />
koha-tmpl/opac-tmpl/bootstrap/en/xslt/MARC21slimUtils.xsl:            <xsl:text> ; </xsl:text>
koha-tmpl/opac-tmpl/bootstrap/en/xslt/MARC21slimUtils.xsl:                <xsl:text> ; </xsl:text><xsl:value-of select="marc:subfield[@code='v']" />

Looking at those results, it is always the series $v that is preceded by a semicolon. Normally, the xslt's do not add such punctuation, but just insert spaces.

So I would recommend here not to call ChopPunctuation but remove those preceding semicolons in the $v display. That seems to be a more consistent approach.

Please feel free to provide counter documentation.
Comment 13 cgresser 2025-08-01 09:25:36 UTC
I'm sorry to jump in again.

At the moment Koha seems to fix globally punctuation "problems" which are not a problem for a cataloguer who does use Marc21 rules! I had previously not mentioned that all 300 fields in modern records seem to chop off the full stop, whereas in MARC21 in RDA in the English-speaking world a full-stop is placed at the end of the 300 field, but only IF there is 490 field, an example of this is:
https://hey.koha.openfifth.net/cgi-bin/koha/opac-detail.pl?biblionumber=98772&
"Description: xvii, 508 pages ; 26 cm"
in OPAC
"300 ## - PHYSICAL DESCRIPTION
Extent	xvii, 508 pages ;
Dimensions	26 cm."
in Marc display.

This might be against an IT professionals or coder's logic, but it is a rule in Marc21 (I don't like it either, but there we are). If the record is AACR2 as opposed to RDA, then I believe that every 300 field should end in a full-stop. Always! [the full-stop removal is less of an issue, as most people looking at our records do not know what should be there, but it is a point in case for Koha globally changing the display of records it doesn't understand the full syntax of!)

I love KOHA a lot, but for Koha globally to chop punctuation off for end of 300 field, or add an extra semi-colon in an 830 series title is madness. Honestly, why do we cataloguers bother to follow the syntax and rules of Marc21 and RDA if Koha then has a global process of chopping punctuation, or adding an extra one.

I don't know how to explain this better, but Koha is trying to fix something with a sledgehammer (the chopping bit), and like an over-excited toddler adding things (the semi-colon), where we human beings do a good job of following syntax. This doesn't make sense.

Sadly, I don't know Koha well-enough to make a call here, but if the leader info suggests that a record is in Marc21 then Koha should stay clear of globally changing how records are displayed, but follow what the cataloguer has provided as Marc.

If some Marc data has "mistakes" in them, either a diligent human being needs to deal with them individually, or if desired with a global change for _their_ catalogue/data. Applying a fix for some data globally, for everyone who uses Koha uses a dictatorial approach which I haven't thought was Koha's intention.
Comment 14 Alexander Wagner 2025-08-04 11:20:28 UTC
(In reply to cgresser from comment #13)

> At the moment Koha seems to fix globally punctuation "problems" which are
> not a problem for a cataloguer who does use Marc21 rules! I had previously
[...]
> If the record is AACR2
> as opposed to RDA, then I believe that every 300 field should end in a
> full-stop. Always!

I think this is one of the examples where you could not (easily) do the punctuation automatically in general. It's a bit along my argument: I perfectly agree that IT people would have written cataloguing rules differently, but as you say, they are as they are.

> [the full-stop removal is less of an issue, as most
> people looking at our records do not know what should be there, but it is a
> point in case for Koha globally changing the display of records it doesn't
> understand the full syntax of!)

IMHO _in general_ one _can't_. There are too many exceptions. Sometimes, it's hard for me to get cataloguing and rules into one sentence. In some areas it sounds more like a list of exceptions. Like German grammar ;)
 
> I love KOHA a lot, but for Koha globally to chop punctuation off for end of
[...]
> adding things (the semi-colon), where we human beings do a good job of
> following syntax. This doesn't make sense.

I'd see a few more nuances here. In Marc there _is_ actually a mechanism to tell if you should add punctuation or not on a per record level. This is said leader (cf. https://www.loc.gov/marc/bibliographic/bdleader.html):

18 - Descriptive cataloging form
    # - Non-ISBD
    a - AACR 2
    c - ISBD punctuation omitted
    i - ISBD punctuation included
    n - Non-ISBD punctuation omitted
    u - Unknown 

I would argue, that if you set the leader 18 to `c` or `n` explicitly, then you as a cataloguer say: "I don't bother about punctuation. Dear computer do it for me as best you can." And it's your cataloguers conscious decision. So, IMHO in these cases Koha should try as best as it can to add punctuation.

OTOH, it's better to leave the stuff _as is_ as our cataloguer told the system "don't touch the punctuation, I already did it".

And if the display is still wrong IMHO it's time to _fix the record_. (By setting the leader appropriately and/or fixing the punctuation.) 
 
> If some Marc data has "mistakes" in them, either a diligent human being
> needs to deal with them individually, or if desired with a global change for
> _their_ catalogue/data.

The Marc-approach seems to me even better than a global default as it is on the _per record_ level.

E.g. one may ingest data from various sources via Z39.50, SRU, harvesting etc. So it may well happen that the source systems approach to punctuation differs. I learned that the union catalogue we use as our main source adds punctuation automatically so "non in the record" all the time, while my impression is that LoC _adds_ the punctuation to the records "all the time".

Now we ingest from LoC if our union catalogue can't deliver => in general I end up with per record rules wrt punctuation. Without obeying the leader my fellow colleagues have to check punctuation in every record and fix it in every other or so. So, I think it would be nice to follow the data and act accordingly.

> Applying a fix for some data globally, for everyone
> who uses Koha uses a dictatorial approach which I haven't thought was Koha's
> intention.

That's a bit harsh.
Comment 15 cgresser 2025-08-04 12:12:33 UTC
(In reply to Alexander Wagner from comment #14)
> (In reply to cgresser from comment #13)
> 
> > At the moment Koha seems to fix globally punctuation "problems" which are
> > not a problem for a cataloguer who does use Marc21 rules! I had previously
> [...]
> > If the record is AACR2
> > as opposed to RDA, then I believe that every 300 field should end in a
> > full-stop. Always!
> 
> I think this is one of the examples where you could not (easily) do the
> punctuation automatically in general. It's a bit along my argument: I
> perfectly agree that IT people would have written cataloguing rules
> differently, but as you say, they are as they are.

Thank you for backing me up on this one, and hopefully you having explained his better than me. If a human being creates catalogue records they might (!) have used the right syntax/punctuation/coding in the Marc record. At least that is a likely possibility, as opposed to Koha believing that the punctuation has to be a certain way, and changing punctuation globally, not in a case by case/record by record basis.

> > [the full-stop removal is less of an issue, as most
> > people looking at our records do not know what should be there, but it is a
> > point in case for Koha globally changing the display of records it doesn't
> > understand the full syntax of!)
> 
> IMHO _in general_ one _can't_. There are too many exceptions. Sometimes,
> it's hard for me to get cataloguing and rules into one sentence. In some
> areas it sounds more like a list of exceptions. Like German grammar ;)

I agree that globally changing punctuation in records which might or might have the correct punctuation is impossible.
  
> > I love KOHA a lot, but for Koha globally to chop punctuation off for end of
> [...]
> > adding things (the semi-colon), where we human beings do a good job of
> > following syntax. This doesn't make sense.
> 
> I'd see a few more nuances here. In Marc there _is_ actually a mechanism to
> tell if you should add punctuation or not on a per record level. This is
> said leader (cf. https://www.loc.gov/marc/bibliographic/bdleader.html):
> 
> 18 - Descriptive cataloging form
>     # - Non-ISBD
>     a - AACR 2
>     c - ISBD punctuation omitted
>     i - ISBD punctuation included
>     n - Non-ISBD punctuation omitted
>     u - Unknown 
> 
> I would argue, that if you set the leader 18 to `c` or `n` explicitly, then
> you as a cataloguer say: "I don't bother about punctuation. Dear computer do
> it for me as best you can." And it's your cataloguers conscious decision.
> So, IMHO in these cases Koha should try as best as it can to add punctuation.

That's a good point. I would never have thought of this! We only ever use a or i - in both cases punctuation _is_ included in our records (I'm aware that some libraries omitt [m]any bits of punctuation, and we add punctuation when we importa such records). Koha shouldn't try to fix the punctuation in records, even if some records don't have the correct punctuation, as a lot _will_ have the right punctuation.
 
> OTOH, it's better to leave the stuff _as is_ as our cataloguer told the
> system "don't touch the punctuation, I already did it".

Yes!
 
> And if the display is still wrong IMHO it's time to _fix the record_. (By
> setting the leader appropriately and/or fixing the punctuation.) 

Indeed. 
 
> > If some Marc data has "mistakes" in them, either a diligent human being
> > needs to deal with them individually, or if desired with a global change for
> > _their_ catalogue/data.
> 
> The Marc-approach seems to me even better than a global default as it is on
> the _per record_ level.

This would be very helpful. 

> E.g. one may ingest data from various sources via Z39.50, SRU, harvesting
> etc. So it may well happen that the source systems approach to punctuation
> differs. I learned that the union catalogue we use as our main source adds
> punctuation automatically so "non in the record" all the time, while my
> impression is that LoC _adds_ the punctuation to the records "all the time".

OCLC's data also adds a lot of stuff automatically, which then the cataloguer can review (thinking of my Library's most-common approach: individual records and physical books, rather than a bundle of records in one go); I'm aware that a lot of libraries get, say, ebook packages, and then the records in big bundle of records. However, if such a bundle shows signs of missing (or unwanted punctuation), it would make more sense for this imported data to be changed, according to local needs, forcing a certain idea of punctuation onto all Koha Marc records.

> Now we ingest from LoC if our union catalogue can't deliver => in general I
> end up with per record rules wrt punctuation. Without obeying the leader my
> fellow colleagues have to check punctuation in every record and fix it in
> every other or so. So, I think it would be nice to follow the data and act
> accordingly.

Yes, I like the idea of, and phrase of "follow the data and act accordingly". To me it seems to be the sound and common sense thing to do. I don't think Koha is doing this, and then some punctuation gets globally removed, and others automatically added. Not great. 

Hopefully, someone else will agree with us, and Koha's global response with regard to punctuation on Marc records can be reviewed. If someone wants to have this punctuation approach applied globally to all their data they should really have to switch it on, as an option, not for every Koha instance to use this.
Comment 16 Bernard 2025-08-06 09:11:19 UTC
As I am not a cataloguer, my understanding of some of the detail here is no doubt hazy. However, I think in a nutshell we are saying we don't want to break well catalogued records but we'd like Koha to try to fix ones that are not, or at least to some degree. Given that the rules may vary by field, it seems to me that using a separate punc.pm as drafted in bugs #21313, #37196 and #37325 is going to be the best way to achieve this and not make the xslt unwieldy. Not that this is a trivial piece of work of course. 

I would favour an approach where the each relevant field is presented to punc.pm. If it is "correct" anyway, nothing would be changed and if it is not, then it would be fixed up and returned. This way we do not need to consider leader 18 at all. The point being, "correct" records would not be broken.

How does that sound?
Comment 17 Alexander Wagner 2025-08-06 10:08:14 UTC
(In reply to Bernard from comment #16)
> As I am not a cataloguer, my understanding of some of the detail here is no
> doubt hazy. However, I think in a nutshell we are saying we don't want to
> break well catalogued records but we'd like Koha to try to fix ones that are
> not, or at least to some degree.

I think this is the gist of it.

BTW: just noticed that in #21303 Caroline stumbled upon the automatic fixing via XSLT. (I knew I already read a bug on this, that's was reason for my initial comment.)

> Given that the rules may vary by field, it
> seems to me that using a separate punc.pm as drafted in bugs #21313, #37196
> and #37325 is going to be the best way to achieve this and not make the xslt
> unwieldy.

Indeed, my first approach was by XSLT, but I was happy to learn from Cait that Mark filters exist and could be used for this, even though right now the filters applied are in a static array in the code. (Still the use case would be general enough to justify a change to the list, I think)

> Not that this is a trivial piece of work of course. 

Agree.

I split it up by fields as I think it would allow for a pragmatic approach adding fields as we stumble upon them. Probably fixing up the XSLT on the way. So Koha could improve over time without blocking everything by a huge overhaul. But maybe there is a better approach?

> I would favour an approach where the each relevant field is presented to
> punc.pm. If it is "correct" anyway, nothing would be changed and if it is
> not, then it would be fixed up and returned. This way we do not need to
> consider leader 18 at all.

I do not yet get how you know if a record is "correct"?

Say, I take a tag and pass it though the automatic punctuation and compare it to the punctuation presented. It could be the same or differ. But what is "correct"? If we treat "the same" as correct, we are implicitly enforcing the rules of `Punctuation.pm` to all records as all those that differ get rewritten and those that are the same already follow the very rules.

I could imagine a system setting, if you want a way to disregard the leader. (Seems you have good reason to distrust it.) Something along the lines:

RewritePunctuation:
  - Never -> do nothing
  - Always -> chop and add
  - Evaluate leader 18 -> respect the leaders value on a per record basis

Looking at your initial issue one could ponder if a general rule makes sense that checks if the same char appears twice. E.g. if the punctuation char is ` ;` one could replace ` ; ;` by ` ;`. But I am not sure if there isn't a strange case, I am not aware of, where something like this could even be "correct".
Comment 18 Bernard 2025-08-07 13:12:07 UTC
I see what you mean about how would you know what is "correct"? I don't have access to the rules which dictate these conventions which makes it tricky to follow. But what it looks like we have ended up with is the conclusion that 

1. a computer can't successfully fix this if the data is not of a standard which follows all the rules. 
2. Even if it is "correct", a computer cannot successfully determine this to ensure it doesn't butcher and "uncorrect" the record.

Given this, and on the basis of the best available solution, it seems to me the best that can be done is an expanded punc.pm, plus the RewritePunctuation syspref you have listed. At least the cataloguer at the institution gets to make the decision based upon local knowledge of their data quality.