Bug 24094

Summary: Authority punctuation mismatch prevents linking to correct records
Product: Koha Reporter: Donna <bwsdonna>
Component: MARC Authority data supportAssignee: Nick Clemens <nick>
Status: CLOSED FIXED QA Contact: Testopia <testopia>
Severity: normal    
Priority: P5 - low CC: 1joynelson, heather_hernandez, katrin.fischer, lucas, martin.renvoize, mkstephens, nick
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=24269
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=33100
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.05.00, 19.11.04, 19.05.09
Attachments: Bug 24094: Unit tests
Bug 24094: Strip trailing spaces and punctuation from authority headings
Bug 24094: Unit tests
Bug 24094: Strip trailing spaces and punctuation from authority headings
Bug 24094: Unit tests
Bug 24094: Strip trailing spaces and punctuation from authority headings

Description Donna 2019-11-22 19:02:17 UTC
In a bibliographic record, the comma preceding the relationship designator, i.e., the comma before $e, is prescribed--it has to be there for the record to conform to RDA and MARC21.  Authority records view this as a different name/etc, so it does not match.  Every, single, correctly cataloged RDA/MARC21 record is going to have a $e preceded by a comma in every authority controlled X00 and X10 field.  Authorities need to disregard/strip trailing commas.
Comment 1 Myka Kennedy Stephens 2019-12-11 23:55:13 UTC
Donna,
Would you clarify which search engine you have enabled when you're seeing this behavior? I am doing some extensive testing with authorities on Elasticsearch and am not seeing this problem with the linker. I also have broader_headings enabled under LinkerOptions.
My experience is that the linker is correctly matching x00 headings with a relator term present *only* when the x00 heading has only $a and $e present. For headings that also include dates ($d) or any other subfields, the linker will not find a match. I'm thinking this is a different bug and will be submitting it after I do some more tests.
Comment 2 Donna 2019-12-12 14:11:14 UTC
(In reply to Myka Kennedy Stephens from comment #1)
> Donna,
> Would you clarify which search engine you have enabled when you're seeing
> this behavior? I am doing some extensive testing with authorities on
> Elasticsearch and am not seeing this problem with the linker. I also have
> broader_headings enabled under LinkerOptions.
> My experience is that the linker is correctly matching x00 headings with a
> relator term present *only* when the x00 heading has only $a and $e present.
> For headings that also include dates ($d) or any other subfields, the linker
> will not find a match. I'm thinking this is a different bug and will be
> submitting it after I do some more tests.

This is on Zebra, Myka.
Comment 3 Nick Clemens 2020-01-31 14:26:50 UTC
Created attachment 98238 [details] [review]
Bug 24094: Unit tests
Comment 4 Nick Clemens 2020-01-31 14:26:52 UTC
Created attachment 98239 [details] [review]
Bug 24094: Strip trailing spaces and punctuation from authority headings

Both when searching for and creating new authorities we need to remove
punctuation that exists in the bibliographic record but does not belong in
the authority record.

For example, a series with a volume contains a semicolon in the bib record,
however, this should not be passed to the authority as the volume is not
included in the authority record.

To test:
 1 - Set AutoCreateAuthorities to 'generate'
 2 - Set BiblioAddsAuthorities to 'true'
 3 - Set CatalogModuleRelink to 'Do'
 4 - Find or create a record with:
    a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author
    an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3.
 5 - Save the records and check the links
 6 - Note punctuation is passed through
 7 - Save again, auth records are created again
 8 - Apply patch
 9 - Save again, new auth records are created again
10 - Check the records, punctuation has been removed
11 - Save again, no more records created.
Comment 5 Myka Kennedy Stephens 2020-02-01 01:55:29 UTC
These patches work as expected in both Zebra and ES for personal names. Applied the patch and saved the record again. It stripped the ending "," before $e in a 100 field and created a new authority record. Re-saving the record again kept the 100 field linked to the same authority record.

Uniform titles are another matter. Applied patch and saved the record again. It stripped the ending ";" before $v in an 830 field and created a new authority record. Re-saving the record again created another new authority record and re-linked the 830 to the new authority record.

I'm also noticing that it is still creating additional authority records for other fields, but not consistently. For example, one record I tried kept subject heading links between saves, but another record did not. Switching between Zebra and ES didn't seem to make a difference.

The bibliographic records I tested came from the Library of Congress Z39.50 server:
#1 - ISBN 9781540960573
#2 - ISBN 9789004130784
#3 - ISBN 9781532653308
Comment 6 Nick Clemens 2020-02-10 15:40:30 UTC
(In reply to Myka Kennedy Stephens from comment #5)
> The bibliographic records I tested came from the Library of Congress Z39.50
> server:
> #1 - ISBN 9781540960573
> #2 - ISBN 9789004130784
> #3 - ISBN 9781532653308

I applied the patches, imported the three records, and saved multiple times - the authorities were generated once, and didn't happen again, everything correctly relinked.

There is the possibility that your linker was set to 'Default' and so not linking because there were duplicates in the system and the Default linker only links if a single record is returned.

Can you retest with 'First' or 'Last' match linker, and or clearing out the duplicates?
Comment 7 Myka Kennedy Stephens 2020-02-12 01:06:24 UTC
Created attachment 98735 [details] [review]
Bug 24094: Unit tests

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Comment 8 Myka Kennedy Stephens 2020-02-12 01:06:28 UTC
Created attachment 98736 [details] [review]
Bug 24094: Strip trailing spaces and punctuation from authority headings

Both when searching for and creating new authorities we need to remove
punctuation that exists in the bibliographic record but does not belong in
the authority record.

For example, a series with a volume contains a semicolon in the bib record,
however, this should not be passed to the authority as the volume is not
included in the authority record.

To test:
 1 - Set AutoCreateAuthorities to 'generate'
 2 - Set BiblioAddsAuthorities to 'true'
 3 - Set CatalogModuleRelink to 'Do'
 4 - Find or create a record with:
    a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author
    an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3.
 5 - Save the records and check the links
 6 - Note punctuation is passed through
 7 - Save again, auth records are created again
 8 - Apply patch
 9 - Save again, new auth records are created again
10 - Check the records, punctuation has been removed
11 - Save again, no more records created.

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Comment 9 Myka Kennedy Stephens 2020-02-12 01:24:47 UTC
(In reply to Myka Kennedy Stephens from comment #8)
> To test:
>  1 - Set AutoCreateAuthorities to 'generate'
>  2 - Set BiblioAddsAuthorities to 'true'
The setting is actually 'allow'
>  3 - Set CatalogModuleRelink to 'Do'
3a - Set LinkerModule to 'First Match'
>  4 - Find or create a record with:
>     a 100 field with a subfield e preceded by a comma: 100 $aBoring, M.
> Eugene M ,$e author
>     an 830 field with a volume preceded by a semicolon: 650$aLord of the
> rings ;$v 3.
>  5 - Save the records and check the links
>  6 - Note punctuation is passed through
>  7 - Save again, auth records are created again
>  8 - Apply patch
>  9 - Save again, new auth records are created again
> 10 - Check the records, punctuation has been removed
> 11 - Save again, no more records created.

[HUGE] CAVEATS - 
1. This patch works optimally with Zebra. Results are mixed with ES. It is looking like ES cannot match headings with any type of punctuation--parentheses, hyphens, periods within headings, etc. Further development work will be needed to address this problem with ES. To observe the behavior, test with ES enabled and import/edit ISBN 9789004130784 which includes several subject headings with internal punctuation and a series title heading with parentheses.
2. This patch will strip any trailing punctuation. This includes a period after a middle initial on a personal name heading if that period is not followed by any other punctuation. It would also include the period at the end of a subject heading that ends with etc., B.C., or any other abbreviation in which a period might need to be preserved.

This feels like a bandage that would hold for a little while but isn't a long-term fix. Bug #11300 holds a lot of promise to be a long-term fix, especially if the linker could search a Z39.50 server for a record during the cataloging process and import it (or prompt to import) if it doesn't exist. Ideally, this would import the record with the correct punctuation for an authority record while also preserving the punctuation in the bib record.

In sum, I'm signing off on this but recognize that it has limitations.
Comment 10 Myka Kennedy Stephens 2020-02-13 03:26:06 UTC
Follow-up and further thoughts:

Re: Caveat #1 - 
I went back and tried this again after making sure that the ES authority mapping on my test box was correct and complete. The linker still failed to link the headings. I am now wondering if bug #24269 could be the root cause of that, rather than a shortcoming on the patch for this bug. I tried to apply the patch for 24269 on top of the patch for this bug, but it failed to apply (I know it has already been signed off, though).

Re: Caveat #2 - 
*If* one wanted to build a list of exceptions of authority punctuation that would need to be kept, here are a few places to start (from an incomplete U.S./Library of Congress perspective):
- etc.
- B.C.
- A.D.
- a period following a single letter (as in an initial)
- Dr.
- Jr.
- Mrs.
- Co.
- Dept.
- Inc.
- hyphen (-)
- closing parenthesis )
- question mark ?
- exclamation point !

If this is the path taken, I strongly recommend this list be expanded to be inclusive of punctuation and abbreviations used in authority records of languages other than American English. It could probably use a few more English exceptions, too.
Comment 11 Martin Renvoize 2020-02-18 19:52:17 UTC
Created attachment 99223 [details] [review]
Bug 24094: Unit tests

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 12 Martin Renvoize 2020-02-18 19:52:20 UTC
Created attachment 99224 [details] [review]
Bug 24094: Strip trailing spaces and punctuation from authority headings

Both when searching for and creating new authorities we need to remove
punctuation that exists in the bibliographic record but does not belong in
the authority record.

For example, a series with a volume contains a semicolon in the bib record,
however, this should not be passed to the authority as the volume is not
included in the authority record.

To test:
 1 - Set AutoCreateAuthorities to 'generate'
 2 - Set BiblioAddsAuthorities to 'true'
 3 - Set CatalogModuleRelink to 'Do'
 4 - Find or create a record with:
    a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author
    an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3.
 5 - Save the records and check the links
 6 - Note punctuation is passed through
 7 - Save again, auth records are created again
 8 - Apply patch
 9 - Save again, new auth records are created again
10 - Check the records, punctuation has been removed
11 - Save again, no more records created.

Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 13 Martin Renvoize 2020-02-18 19:52:55 UTC
Works well, QA scripts are happy and it's got unit tests.

Passing QA
Comment 14 Martin Renvoize 2020-02-19 11:34:31 UTC
Nice work everyone!

Pushed to master for 20.05
Comment 15 Joy Nelson 2020-03-13 00:36:55 UTC
Pushed to 19.11.x branch for 19.11.04
Comment 16 Lucas Gass 2020-03-18 21:57:47 UTC
backported to 19.05.x for 19.05.09