|
Description
Donna
2019-11-22 19:02:17 UTC
Donna, Would you clarify which search engine you have enabled when you're seeing this behavior? I am doing some extensive testing with authorities on Elasticsearch and am not seeing this problem with the linker. I also have broader_headings enabled under LinkerOptions. My experience is that the linker is correctly matching x00 headings with a relator term present *only* when the x00 heading has only $a and $e present. For headings that also include dates ($d) or any other subfields, the linker will not find a match. I'm thinking this is a different bug and will be submitting it after I do some more tests. (In reply to Myka Kennedy Stephens from comment #1) > Donna, > Would you clarify which search engine you have enabled when you're seeing > this behavior? I am doing some extensive testing with authorities on > Elasticsearch and am not seeing this problem with the linker. I also have > broader_headings enabled under LinkerOptions. > My experience is that the linker is correctly matching x00 headings with a > relator term present *only* when the x00 heading has only $a and $e present. > For headings that also include dates ($d) or any other subfields, the linker > will not find a match. I'm thinking this is a different bug and will be > submitting it after I do some more tests. This is on Zebra, Myka. Created attachment 98238 [details] [review] Bug 24094: Unit tests Created attachment 98239 [details] [review] Bug 24094: Strip trailing spaces and punctuation from authority headings Both when searching for and creating new authorities we need to remove punctuation that exists in the bibliographic record but does not belong in the authority record. For example, a series with a volume contains a semicolon in the bib record, however, this should not be passed to the authority as the volume is not included in the authority record. To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' 3 - Set CatalogModuleRelink to 'Do' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created. These patches work as expected in both Zebra and ES for personal names. Applied the patch and saved the record again. It stripped the ending "," before $e in a 100 field and created a new authority record. Re-saving the record again kept the 100 field linked to the same authority record. Uniform titles are another matter. Applied patch and saved the record again. It stripped the ending ";" before $v in an 830 field and created a new authority record. Re-saving the record again created another new authority record and re-linked the 830 to the new authority record. I'm also noticing that it is still creating additional authority records for other fields, but not consistently. For example, one record I tried kept subject heading links between saves, but another record did not. Switching between Zebra and ES didn't seem to make a difference. The bibliographic records I tested came from the Library of Congress Z39.50 server: #1 - ISBN 9781540960573 #2 - ISBN 9789004130784 #3 - ISBN 9781532653308 (In reply to Myka Kennedy Stephens from comment #5) > The bibliographic records I tested came from the Library of Congress Z39.50 > server: > #1 - ISBN 9781540960573 > #2 - ISBN 9789004130784 > #3 - ISBN 9781532653308 I applied the patches, imported the three records, and saved multiple times - the authorities were generated once, and didn't happen again, everything correctly relinked. There is the possibility that your linker was set to 'Default' and so not linking because there were duplicates in the system and the Default linker only links if a single record is returned. Can you retest with 'First' or 'Last' match linker, and or clearing out the duplicates? Created attachment 98735 [details] [review] Bug 24094: Unit tests Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> Created attachment 98736 [details] [review] Bug 24094: Strip trailing spaces and punctuation from authority headings Both when searching for and creating new authorities we need to remove punctuation that exists in the bibliographic record but does not belong in the authority record. For example, a series with a volume contains a semicolon in the bib record, however, this should not be passed to the authority as the volume is not included in the authority record. To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' 3 - Set CatalogModuleRelink to 'Do' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created. Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> (In reply to Myka Kennedy Stephens from comment #8) > To test: > 1 - Set AutoCreateAuthorities to 'generate' > 2 - Set BiblioAddsAuthorities to 'true' The setting is actually 'allow' > 3 - Set CatalogModuleRelink to 'Do' 3a - Set LinkerModule to 'First Match' > 4 - Find or create a record with: > a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. > Eugene M ,$e author > an 830 field with a volume preceded by a semicolon: 650$aLord of the > rings ;$v 3. > 5 - Save the records and check the links > 6 - Note punctuation is passed through > 7 - Save again, auth records are created again > 8 - Apply patch > 9 - Save again, new auth records are created again > 10 - Check the records, punctuation has been removed > 11 - Save again, no more records created. [HUGE] CAVEATS - 1. This patch works optimally with Zebra. Results are mixed with ES. It is looking like ES cannot match headings with any type of punctuation--parentheses, hyphens, periods within headings, etc. Further development work will be needed to address this problem with ES. To observe the behavior, test with ES enabled and import/edit ISBN 9789004130784 which includes several subject headings with internal punctuation and a series title heading with parentheses. 2. This patch will strip any trailing punctuation. This includes a period after a middle initial on a personal name heading if that period is not followed by any other punctuation. It would also include the period at the end of a subject heading that ends with etc., B.C., or any other abbreviation in which a period might need to be preserved. This feels like a bandage that would hold for a little while but isn't a long-term fix. Bug #11300 holds a lot of promise to be a long-term fix, especially if the linker could search a Z39.50 server for a record during the cataloging process and import it (or prompt to import) if it doesn't exist. Ideally, this would import the record with the correct punctuation for an authority record while also preserving the punctuation in the bib record. In sum, I'm signing off on this but recognize that it has limitations. Follow-up and further thoughts: Re: Caveat #1 - I went back and tried this again after making sure that the ES authority mapping on my test box was correct and complete. The linker still failed to link the headings. I am now wondering if bug #24269 could be the root cause of that, rather than a shortcoming on the patch for this bug. I tried to apply the patch for 24269 on top of the patch for this bug, but it failed to apply (I know it has already been signed off, though). Re: Caveat #2 - *If* one wanted to build a list of exceptions of authority punctuation that would need to be kept, here are a few places to start (from an incomplete U.S./Library of Congress perspective): - etc. - B.C. - A.D. - a period following a single letter (as in an initial) - Dr. - Jr. - Mrs. - Co. - Dept. - Inc. - hyphen (-) - closing parenthesis ) - question mark ? - exclamation point ! If this is the path taken, I strongly recommend this list be expanded to be inclusive of punctuation and abbreviations used in authority records of languages other than American English. It could probably use a few more English exceptions, too. Created attachment 99223 [details] [review] Bug 24094: Unit tests Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Created attachment 99224 [details] [review] Bug 24094: Strip trailing spaces and punctuation from authority headings Both when searching for and creating new authorities we need to remove punctuation that exists in the bibliographic record but does not belong in the authority record. For example, a series with a volume contains a semicolon in the bib record, however, this should not be passed to the authority as the volume is not included in the authority record. To test: 1 - Set AutoCreateAuthorities to 'generate' 2 - Set BiblioAddsAuthorities to 'true' 3 - Set CatalogModuleRelink to 'Do' 4 - Find or create a record with: a 100 field with a subfield e preceded by a comma: 100 $aBoring, M. Eugene M ,$e author an 830 field with a volume preceded by a semicolon: 650$aLord of the rings ;$v 3. 5 - Save the records and check the links 6 - Note punctuation is passed through 7 - Save again, auth records are created again 8 - Apply patch 9 - Save again, new auth records are created again 10 - Check the records, punctuation has been removed 11 - Save again, no more records created. Signed-off-by: Myka Kennedy Stephens <mkstephens@lancasterseminary.edu> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> Works well, QA scripts are happy and it's got unit tests. Passing QA Nice work everyone! Pushed to master for 20.05 Pushed to 19.11.x branch for 19.11.04 backported to 19.05.x for 19.05.09 |