Bug 33100

Summary: Authority linking doesn't work for bib headings ending in two or more punctuation characters
Product: Koha Reporter: Phil Ringnalda <phil>
Component: CatalogingAssignee: Nick Clemens (kidclamp) <nick>
Status: RESOLVED FIXED QA Contact: Martin Renvoize (ashimema) <martin.renvoize>
Severity: major    
Priority: P5 - low CC: arthur.suzuki, flyingendpaper, lucas, m.de.rooy, martin.renvoize, nick
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24094
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24833
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
23.05.00,22.11.04,22.05.12, 21.11.20
Circulation function:
Bug Depends on:    
Bug Blocks: 33159    
Attachments: Bug 33100: Unit test
Bug 33100: Remove all punctuation at end of record
Bug 33100: Unit test
Bug 33100: Remove all punctuation at end of record
Bug 33100: Unit test
Bug 33100: Remove all punctuation at end of record
Bug 33100: (QA follow-up) Trivial QA fixes

Description Phil Ringnalda 2023-03-01 05:51:53 UTC
At the time of bug 24094 this apparently worked, but now in both my production 22.05 instance and a master sandbox, while Doe, John A. in a bib record will link to Doe, John A. in an authority record, Doe, John A., (as it will be in virtually every RDA record, since you are supposed to have a $e relator for personal names, and you are supposed to have a comma before the $e) will not link to the Doe, John A. authority record.

That's a barely noticeable thing unless you actually track your unlinked authorites, but it's very noticeable if you use AutoCreateAuthorites and the first-match linker. In that case, every time you add a new Box, C. J. item, the linker strips off the comma from $aBox, C. J.,$eauthor., declares that your existing authority record for Box, C. J. doesn't match, and AutoCreateAuthorities also strips off the comma, creates a new duplicate Box, C. J. record, and then does it again the next time you save the record, and then does it again when you update the authority record for the series heading because it changed, and that triggers relinking the whole bib record, then again any time you edit the record or any authority linked to it.

Two maybe-interesting exceptions, from when I started out looking at all the punctuation that _get_search_form in C4/Heading/MARC21.pm trims from the end of a bib heading to attempt to make it searchable, and the not-quite-the-same things that AutoCreateAuthorities strips before creating what it hopes will be an authority heading that will actually still match during the next linking:

Doe, John A-- will autocreate an authority record which is exactly that, because autocreate's set of punctuation lack only that one character from _get_search_heading's regex, but, even though _get_search_heading will turn that into a search for Doe, John A- during a relink, that will match the -- authority form. The same is true for other characters: if you edit the authority record for Doe, John A. and make it Doe, John A., then that will match a bib record with Doe, John A., and will also match a bib record with Doe, John A.! or Doe, John A./ or Doe, John A.^ which isn't even in the set of stripped punctuation so the search should be for Doe, John A.^ and that still matches Doe, John A.,

Doe, John A=, will match an authority record for Doe, John A with no punctuation, though there's no similar magic for Doe, John,= where _get_search_heading has stripped off the equal sign, making me think at least that bit of magic is happening in the search engine or the query prep.
Comment 1 Phil Ringnalda 2023-03-07 18:57:33 UTC
May wind up invalid, I forgot that the fallout from bug 30280 means you can't actually test linking on master/22.11, because almost everything fails to link and thus autocreates duplicates.
Comment 2 Nick Clemens (kidclamp) 2023-03-07 20:09:38 UTC
Created attachment 147885 [details] [review]
Bug 33100: Unit test
Comment 3 Nick Clemens (kidclamp) 2023-03-07 20:09:40 UTC
Created attachment 147886 [details] [review]
Bug 33100: Remove all punctuation at end of record

This could potentially be overkill - if you have two records that only
differ in ending punctuation, you might get a false result - I don't
know how likely that is ( as I have mentioned before '!!!' is a band )

This seems a reasonable fix

To test:
1 - Set sysprefs as in bug 33159
2 - Edit a record, biblionumebr 3 is fine
3 - Add a comma to end of author subfield a
4 - Save and note relinked
5 - Apply patch, restart all
6 - Save again
7 - Linked to original record
Comment 4 Phil Ringnalda 2023-03-07 20:38:50 UTC
Whether or not you can have collisions depends on what heading rules you catalog under, but at least for Library of Congress/NACO rules you're pretty safe: a heading has to be unique after https://www.loc.gov/aba/pcc/naco/normrule-2.html#a is applied, which removes or blanks most punctuation. While Foo&, Foo#, and Foo are validly three different names (which is fine, we don't strip & or #), Foo-, Foo., Foo:, Foo=, Foo;, Foo%, Foo/, and Foo are all the same name so only one of them can exist.

Comma is a little bit fun, because normalization says that the "first comma in $a  is retained" but doesn't say "only if there is something following it" so technically if there's already someone named "Foo" it would be valid to have someone else named "Foo," but at least so far people giving themselves Cher-like single names don't seem very fond of having a trailing comma as part of their actual name.
Comment 5 Nick Clemens (kidclamp) 2023-03-09 20:42:18 UTC
Created attachment 148030 [details] [review]
Bug 33100: Unit test
Comment 6 Nick Clemens (kidclamp) 2023-03-09 20:42:20 UTC
Created attachment 148031 [details] [review]
Bug 33100: Remove all punctuation at end of record

This could potentially be overkill - if you have two records that only
differ in ending punctuation, you might get a false result - I don't
know how likely that is ( as I have mentioned before '!!!' is a band )

This seems a reasonable fix

To test:
1 - Set sysprefs as in bug 33159
2 - Edit a record, biblionumebr 3 is fine
3 - Add a comma to end of author subfield a
4 - Save and note relinked
5 - Apply patch, restart all
6 - Save again
7 - Linked to original record
Comment 7 Phil Ringnalda 2023-03-10 18:56:25 UTC
I can only test locally with Zebra, because I've never gotten the instructions to make ElasticSearch work on macOS to actually work.

With Zebra, 22.05.x does not exhibit this bug: after you add the comma, the heading still links to the original authority

With Zebra, master @ 7a829606a46f5d3c2056ce9da5134ce293d584ad (the revision before bug 30280 broke linking names) does not exhibit this bug

With Zebra, HEAD plus the patches from bug 33159 to allow linking names to work again does not exhibit this bug

With ElasticSearch on a sandbox, HEAD plus the patches from bug 33159 to allow linking names does not exhibit this bug

I'd like to know whether ElasticSearch on 22.05.x, ElasticSearch @ 7a829606a46f5d3c2056ce9da5134ce293d584ad, and ElasticSearch on 22.11.x exhibit this bug, but I don't have any way to find out myself.
Comment 8 Martin Renvoize (ashimema) 2023-03-15 16:11:47 UTC
Created attachment 148225 [details] [review]
Bug 33100: Unit test

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 9 Martin Renvoize (ashimema) 2023-03-15 16:11:50 UTC
Created attachment 148226 [details] [review]
Bug 33100: Remove all punctuation at end of record

This could potentially be overkill - if you have two records that only
differ in ending punctuation, you might get a false result - I don't
know how likely that is ( as I have mentioned before '!!!' is a band )

This seems a reasonable fix

To test:
1 - Set sysprefs as in bug 33159
2 - Edit a record, biblionumebr 3 is fine
3 - Add a comma to end of author subfield a
4 - Save and note relinked
5 - Apply patch, restart all
6 - Save again
7 - Linked to original record

Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 10 Martin Renvoize (ashimema) 2023-03-15 16:13:05 UTC
The code change is nice and clear here and in testing it's all working as expected for me.  We're also nicely covered by unit tests.

QA scripts are happy too, so I'm going to straight PQA this one.
Comment 11 Tomás Cohen Arazi (tcohen) 2023-03-17 12:55:02 UTC
Created attachment 148337 [details] [review]
Bug 33100: (QA follow-up) Trivial QA fixes

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 12 Tomás Cohen Arazi (tcohen) 2023-03-17 13:00:08 UTC
Pushed to master for 23.05.

Nice work everyone, thanks!
Comment 13 Matt Blenkinsop 2023-03-17 17:21:04 UTC
Nice work everyone!

Pushed to stable for 22.11.x
Comment 14 Lucas Gass (lukeg) 2023-04-21 16:54:55 UTC
Backported to 22.05.x for upcoming 22.05.12
Comment 15 Arthur Suzuki 2023-05-03 18:33:17 UTC
applied to 21.11.x for 21.11.20