Currently, Koha linker does not pay attention to the diacritics while linking--the decision about linking is taken based on the results of search in the search engine (be it Zebra or Elasticsearch) which are diacritics insensitive. As a consequence, "Brückner, Karl" could be erroneously linked to "Bruckner, Karl", in case when "Brückner, Karl" did not exist in the authority file, or no link would be made in case when both "Brückner, Karl" and "Bruckner, Karl" existed in the authority file and LinkerModule is set to 'default'.
Created attachment 175626 [details] [review] Bug 38729: Linker should consider diacritics Currently, Koha linker does not pay attention to the diacritics while linking--the decision about linking is taken based on the results of search in the search engine (be it Zebra or Elasticsearch) which are diacritics insensitive. As a consequence, "Brückner, Karl" could be erroneously linked to "Bruckner, Karl", in case when "Brückner, Karl" did not exist in the authority file, or no link would be made in case when both "Brückner, Karl" and "Bruckner, Karl" existed in the authority file and LinkerModule is set to 'default'. Test plan: ========== 1. Make sure that LinkerModule system preference is set to default. 2. Create an authority record of type PERSO_NAME with a name containing diacritics (e.g. "Brückner, Karl" , or "Rąkowski, Albin" etc.). 3. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) that name without diacritics. 4. Click "Link authorities automatically" -- the two different names will be linked, which is wrong. 5. Create a second authority record with name form exactly from the bibliographic record (without diacritics). 6. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) the name without diacritics. 7. Click "Link authorities automatically" -- you should get a message "More than one local match found. Possibly a duplicate authority!". 8. Apply the patch ; restart_all. 9. Choose another name pair with and without diacritics. Repeat p. 3-4. The bibliographic field should not be linked this time. 10. With the new pair, repeat p. 5.-7. The bibliographic field should now be correctly linked. Sponsored-by: Ignatianum University in Cracow
Created attachment 175627 [details] [review] Bug 38729: Unit tests
Lest the NACO authority normalization rules in http://www.loc.gov/aba/pcc/naco/normrule-2.html come up, note footnote 4 in the report on which they are based, http://www.loc.gov/aba/pcc/naco/documents/SCA_PccNormalization_Final_revised.pdf which says that one of the things they explicitly do not cover is "matching access fields in authority records to variable fields in bibliographic records." With a hypothetical UseNACONormalization pref turned on, you should be warned when saving an authority record for "Brückner, Karl" that it is an unacceptable duplicate of "Bruckner, Karl," but it shouldn't have anything to do with linking.
(In reply to Phil Ringnalda from comment #3) > Lest the NACO authority normalization rules in > http://www.loc.gov/aba/pcc/naco/normrule-2.html come up, note footnote 4 in > the report on which they are based, > http://www.loc.gov/aba/pcc/naco/documents/SCA_PccNormalization_Final_revised. > pdf which says that one of the things they explicitly do not cover is > "matching access fields in authority records to variable fields in > bibliographic records." With a hypothetical UseNACONormalization pref turned > on, you should be warned when saving an authority record for "Brückner, > Karl" that it is an unacceptable duplicate of "Bruckner, Karl," but it > shouldn't have anything to do with linking. I must say, I have little idea about PCC regulation. Maybe forming two authorities for two different person like "Brückner, Karl" and "Bruckner, Karl" would not be valid in PCC NACO context. But we should accept the fact that not every library in the world is obliged to follow PCC NACO guidelines. Some real world examples from the Polish context, where letters pairs like 'o' and 'ó', 'a' and 'ą', 'z' and 'ż' are completely different letters: https://viaf.org/viaf/search?query=local.names%20all%20%22Jamroz%2C%20Agnieszka.%22&sortKeys=holdingscount&recordSchema=BriefVIAF Jamroz, Jamroż and Jamróz are theree different surnames (with the same first name Agnieszka). Another one: https://viaf.org/viaf/search?query=local.names%20all%20%22Mrozek%2C%20Sebastian.%22&sortKeys=holdingscount&recordSchema=BriefVIAF Mrózek, Mrozek and Mrożek (with the same name first name Sławomir) And yet another one: https://viaf.org/viaf/search?query=local.names%20all%20%22Klos%2C%20Adam.%22&sortKeys=holdingscount&recordSchema=BriefVIAF Kłos, Klos, Kłóś Adam to point only a few examples. So this is definitely a question of linking--choosing the right authority record for a given heading in the bibliographic record. IMO, the proposed tuning of the linker should in no way hurt the libraries following PCC NACO rules, and at the same time will make it possible for other libraries to use the default Koha linker without local tweaks.
I like the example in bug 33401 (which is probably a duplicate of this, despite having talked about autocreate rather than about linking): Koha can't tell the difference between a Swedish widow, Änkor, and a Swedish duck, Ankor.
(In reply to Phil Ringnalda from comment #5) > I like the example in bug 33401 (which is probably a duplicate of this, > despite having talked about autocreate rather than about linking): Koha > can't tell the difference between a Swedish widow, Änkor, and a Swedish > duck, Ankor. Personally, I do not like auto created authorities (among others, Koha does not pay attention to the correct construction of 008 field, and: why not to use the official authority files), but we could have a look at the example after solving this real issue. But you are right, these are like two sides of the coin. I presume that solving this bug will stop Koha to equate Å with Ä, and with A also while creating authorities (since nothing would be found for 'Ankor' even if 'Änkor' existed--the new authority would be auto created).
This is an interesting one for sure, and it seems like a very real problem. However, I think that you need to make this feature optional with your current implementation. Your patch would unexpectedly change existing behaviour that people rely upon. For instance, there are lots of cases where your bib heading doesn't perfectly match the authority (e.g. punctuation difference, minor spelling difference in one of the words in the heading, whitespace difference, etc), but you want it to still match and use the authorized form.
Also, I think that your intention here might be locale specific, so there might be a better way of doing this change. Take a look at bug 36947 where I use a locale-specific sort for Elasticsearch facets. Consider the functionality available in Unicode::Collate and Unicode::Collate::Locale. You might consider using $Collator->eq($auth_heading->search_form,$self->search_for) to have a more nuanced comparison. I'd love to hear from some French and Spanish libraries about what they'd like here, because I think diacritics get ignored in those locales during collation. However, I think for German u and ü are treated as separate letters for collation, although I haven't verified. Anyway, not saying you have to use the Unicode modules for this. It's just something to consider. -- Another thought I had was perhaps you can configure your Elasticsearch differently. Consider this email regarding Arabic libraries: https://lists.katipo.co.nz/public/koha/2024-August/060968.html. While there isn't a Polish analyser out of the box, there is a "Stempel Polish analysis plugin". You'd need to update a number of Koha ES configuration files, but it could be doable. -- Note the above aren't requirements. Just suggestions that might help you.
Created attachment 175640 [details] [review] Bug 38729: Unit tests
Created attachment 175644 [details] [review] Bug 38729: Linker should consider diacritics Currently, Koha linker does not pay attention to the diacritics while linking--the decision about linking is taken based on the results of search in the search engine (be it Zebra or Elasticsearch) which are diacritics insensitive. As a consequence, "Brückner, Karl" could be erroneously linked to "Bruckner, Karl", in case when "Brückner, Karl" did not exist in the authority file, or no link would be made in case when both "Brückner, Karl" and "Bruckner, Karl" existed in the authority file and LinkerModule is set to 'default'. Test plan: ========== 1. Make sure that LinkerModule system preference is set to default. 2. Create an authority record of type PERSO_NAME with a name containing diacritics (e.g. "Brückner, Karl" , or "Rąkowski, Albin" etc.). 3. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) that name without diacritics. 4. Click "Link authorities automatically" -- the two different names will be linked, which is wrong. 5. Create a second authority record with name form exactly from the bibliographic record (without diacritics). 6. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) the name without diacritics. 7. Click "Link authorities automatically" -- you should get a message "More than one local match found. Possibly a duplicate authority!". 8. Apply the patch ; restart_all. 9. Choose another name pair with and without diacritics. Repeat p. 3-4. The bibliographic field should not be linked this time. 10. With the new pair, repeat p. 5.-7. The bibliographic field should now be correctly linked. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Created attachment 175645 [details] [review] Bug 38729: Unit tests Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>