Currently, Koha linker does not pay attention to the diacritics while linking--the decision about linking is taken based on the results of search in the search engine (be it Zebra or Elasticsearch) which are diacritics insensitive. As a consequence, "Brückner, Karl" could be erroneously linked to "Bruckner, Karl", in case when "Brückner, Karl" did not exist in the authority file, or no link would be made in case when both "Brückner, Karl" and "Bruckner, Karl" existed in the authority file and LinkerModule is set to 'default'.
Created attachment 175626 [details] [review] Bug 38729: Linker should consider diacritics Currently, Koha linker does not pay attention to the diacritics while linking--the decision about linking is taken based on the results of search in the search engine (be it Zebra or Elasticsearch) which are diacritics insensitive. As a consequence, "Brückner, Karl" could be erroneously linked to "Bruckner, Karl", in case when "Brückner, Karl" did not exist in the authority file, or no link would be made in case when both "Brückner, Karl" and "Bruckner, Karl" existed in the authority file and LinkerModule is set to 'default'. Test plan: ========== 1. Make sure that LinkerModule system preference is set to default. 2. Create an authority record of type PERSO_NAME with a name containing diacritics (e.g. "Brückner, Karl" , or "Rąkowski, Albin" etc.). 3. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) that name without diacritics. 4. Click "Link authorities automatically" -- the two different names will be linked, which is wrong. 5. Create a second authority record with name form exactly from the bibliographic record (without diacritics). 6. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) the name without diacritics. 7. Click "Link authorities automatically" -- you should get a message "More than one local match found. Possibly a duplicate authority!". 8. Apply the patch ; restart_all. 9. Choose another name pair with and without diacritics. Repeat p. 3-4. The bibliographic field should not be linked this time. 10. With the new pair, repeat p. 5.-7. The bibliographic field should now be correctly linked. Sponsored-by: Ignatianum University in Cracow
Created attachment 175627 [details] [review] Bug 38729: Unit tests
Lest the NACO authority normalization rules in http://www.loc.gov/aba/pcc/naco/normrule-2.html come up, note footnote 4 in the report on which they are based, http://www.loc.gov/aba/pcc/naco/documents/SCA_PccNormalization_Final_revised.pdf which says that one of the things they explicitly do not cover is "matching access fields in authority records to variable fields in bibliographic records." With a hypothetical UseNACONormalization pref turned on, you should be warned when saving an authority record for "Brückner, Karl" that it is an unacceptable duplicate of "Bruckner, Karl," but it shouldn't have anything to do with linking.
(In reply to Phil Ringnalda from comment #3) > Lest the NACO authority normalization rules in > http://www.loc.gov/aba/pcc/naco/normrule-2.html come up, note footnote 4 in > the report on which they are based, > http://www.loc.gov/aba/pcc/naco/documents/SCA_PccNormalization_Final_revised. > pdf which says that one of the things they explicitly do not cover is > "matching access fields in authority records to variable fields in > bibliographic records." With a hypothetical UseNACONormalization pref turned > on, you should be warned when saving an authority record for "Brückner, > Karl" that it is an unacceptable duplicate of "Bruckner, Karl," but it > shouldn't have anything to do with linking. I must say, I have little idea about PCC regulation. Maybe forming two authorities for two different person like "Brückner, Karl" and "Bruckner, Karl" would not be valid in PCC NACO context. But we should accept the fact that not every library in the world is obliged to follow PCC NACO guidelines. Some real world examples from the Polish context, where letters pairs like 'o' and 'ó', 'a' and 'ą', 'z' and 'ż' are completely different letters: https://viaf.org/viaf/search?query=local.names%20all%20%22Jamroz%2C%20Agnieszka.%22&sortKeys=holdingscount&recordSchema=BriefVIAF Jamroz, Jamroż and Jamróz are theree different surnames (with the same first name Agnieszka). Another one: https://viaf.org/viaf/search?query=local.names%20all%20%22Mrozek%2C%20Sebastian.%22&sortKeys=holdingscount&recordSchema=BriefVIAF Mrózek, Mrozek and Mrożek (with the same name first name Sławomir) And yet another one: https://viaf.org/viaf/search?query=local.names%20all%20%22Klos%2C%20Adam.%22&sortKeys=holdingscount&recordSchema=BriefVIAF Kłos, Klos, Kłóś Adam to point only a few examples. So this is definitely a question of linking--choosing the right authority record for a given heading in the bibliographic record. IMO, the proposed tuning of the linker should in no way hurt the libraries following PCC NACO rules, and at the same time will make it possible for other libraries to use the default Koha linker without local tweaks.
I like the example in bug 33401 (which is probably a duplicate of this, despite having talked about autocreate rather than about linking): Koha can't tell the difference between a Swedish widow, Änkor, and a Swedish duck, Ankor.
(In reply to Phil Ringnalda from comment #5) > I like the example in bug 33401 (which is probably a duplicate of this, > despite having talked about autocreate rather than about linking): Koha > can't tell the difference between a Swedish widow, Änkor, and a Swedish > duck, Ankor. Personally, I do not like auto created authorities (among others, Koha does not pay attention to the correct construction of 008 field, and: why not to use the official authority files), but we could have a look at the example after solving this real issue. But you are right, these are like two sides of the coin. I presume that solving this bug will stop Koha to equate Å with Ä, and with A also while creating authorities (since nothing would be found for 'Ankor' even if 'Änkor' existed--the new authority would be auto created).
This is an interesting one for sure, and it seems like a very real problem. However, I think that you need to make this feature optional with your current implementation. Your patch would unexpectedly change existing behaviour that people rely upon. For instance, there are lots of cases where your bib heading doesn't perfectly match the authority (e.g. punctuation difference, minor spelling difference in one of the words in the heading, whitespace difference, etc), but you want it to still match and use the authorized form.
Also, I think that your intention here might be locale specific, so there might be a better way of doing this change. Take a look at bug 36947 where I use a locale-specific sort for Elasticsearch facets. Consider the functionality available in Unicode::Collate and Unicode::Collate::Locale. You might consider using $Collator->eq($auth_heading->search_form,$self->search_for) to have a more nuanced comparison. I'd love to hear from some French and Spanish libraries about what they'd like here, because I think diacritics get ignored in those locales during collation. However, I think for German u and ü are treated as separate letters for collation, although I haven't verified. Anyway, not saying you have to use the Unicode modules for this. It's just something to consider. -- Another thought I had was perhaps you can configure your Elasticsearch differently. Consider this email regarding Arabic libraries: https://lists.katipo.co.nz/public/koha/2024-August/060968.html. While there isn't a Polish analyser out of the box, there is a "Stempel Polish analysis plugin". You'd need to update a number of Koha ES configuration files, but it could be doable. -- Note the above aren't requirements. Just suggestions that might help you.
Created attachment 175640 [details] [review] Bug 38729: Unit tests
Created attachment 175644 [details] [review] Bug 38729: Linker should consider diacritics Currently, Koha linker does not pay attention to the diacritics while linking--the decision about linking is taken based on the results of search in the search engine (be it Zebra or Elasticsearch) which are diacritics insensitive. As a consequence, "Brückner, Karl" could be erroneously linked to "Bruckner, Karl", in case when "Brückner, Karl" did not exist in the authority file, or no link would be made in case when both "Brückner, Karl" and "Bruckner, Karl" existed in the authority file and LinkerModule is set to 'default'. Test plan: ========== 1. Make sure that LinkerModule system preference is set to default. 2. Create an authority record of type PERSO_NAME with a name containing diacritics (e.g. "Brückner, Karl" , or "Rąkowski, Albin" etc.). 3. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) that name without diacritics. 4. Click "Link authorities automatically" -- the two different names will be linked, which is wrong. 5. Create a second authority record with name form exactly from the bibliographic record (without diacritics). 6. Edit an existing or create a new bibliographic record and put in a personal name field (100, 600, 700) the name without diacritics. 7. Click "Link authorities automatically" -- you should get a message "More than one local match found. Possibly a duplicate authority!". 8. Apply the patch ; restart_all. 9. Choose another name pair with and without diacritics. Repeat p. 3-4. The bibliographic field should not be linked this time. 10. With the new pair, repeat p. 5.-7. The bibliographic field should now be correctly linked. Sponsored-by: Ignatianum University in Cracow Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
Created attachment 175645 [details] [review] Bug 38729: Unit tests Signed-off-by: Roman Dolny <roman.dolny@jezuici.pl>
(In reply to David Cook from comment #7) > This is an interesting one for sure, and it seems like a very real problem. > > However, I think that you need to make this feature optional with your > current implementation. Your patch would unexpectedly change existing > behaviour that people rely upon. > > For instance, there are lots of cases where your bib heading doesn't > perfectly match the authority (e.g. punctuation difference, minor spelling > difference in one of the words in the heading, whitespace difference, etc), > but you want it to still match and use the authorized form. Good points. Janusz, could you address that concern please? Changing status to reflect need for feedback.
(In reply to Marcel de Rooy from comment #12) > (In reply to David Cook from comment #7) > > This is an interesting one for sure, and it seems like a very real problem. > > > > However, I think that you need to make this feature optional with your > > current implementation. Your patch would unexpectedly change existing > > behaviour that people rely upon. > > > > For instance, there are lots of cases where your bib heading doesn't > > perfectly match the authority (e.g. punctuation difference, minor spelling > > difference in one of the words in the heading, whitespace difference, etc), > > but you want it to still match and use the authorized form. > > Good points. Janusz, could you address that concern please? > Changing status to reflect need for feedback. Well, lets try. The main aim with this patch is to distinguish between Latin-based letters that should not be equated with each other. I am aware that in English alphabet there are just 26 basic letters and may be that from an anglophone point of view 'Å', 'Ä', 'Ą', 'Á' (to name only a few letters based on Latin letter 'A') all equal to 'A', but there do not -- they are separate letters in the respective alphabets (Swedish, German, Polish, Hungarian...): https://en.wikipedia.org/wiki/Swedish_alphabet, https://en.wikipedia.org/wiki/Hungarian_alphabet , etc. Also, it is not a local issue IMO but rather a general one. It will be an issue in every catalogue collecting international literature. You should definitely distinguish names like 'Jamroz', 'Jamroż' and 'Jamróz'. I agree it will not occur in one in ten cases, but this should not be a reason to ignore the issue. The fact that this is relatively rare is an explanation for why it went undetected for so long. (BTW this issue reveals itself more easily with large scale catalogues, with several hundred thousand or several million records.) At the same time, traditionally, when searching, we expect to find all three names ('Jamroz', 'Jamroż' and 'Jamróz') by searching without diacritical marks, i.e. 'Jamroz' (important especially for those who do not use accrual keyboards). And not just in the Polish catalogue, but in every catalogue. So Elasticsearch works IMO correctly with current default settings. As for the inaccuracy of the record, as a librarian I prefer to have a controlled field unlinked to the authority record than linked incorrectly and then perhaps modified incorrectly as a consequence (e.g. Änkor linked to Ankor will become Ankor when the Ankor authority record is first edited, cf. Bug 33401). Finally, regarding punctuation, here I am comparing the search_form, which is constructed by _get_search_heading. This method, among other things, removes the parentheses and punctuation from the end of each subfield, and so largely standardizes the notation, bypassing some of the possible problems of inaccurate notation.
(In reply to Janusz Kaczmarek from comment #13) > Finally, regarding punctuation, here I am comparing the search_form, which > is constructed by _get_search_heading. This method, among other things, > removes the parentheses and punctuation from the end of each subfield, and > so largely standardizes the notation, bypassing some of the possible > problems of inaccurate notation. And interestingly, even without current patch default linker will not link a heading if there is a difference in the internal punctuation (inside a subfield) but will will link when there is a difference in letters. To illustrate: with ktd standard data, with ES, 1. try to link: Heylin Clinton. [no comma] --> no result 2. try to link: Heylin, Clinton. [two spaces after a comma] --> no result 3. try to link: Heylin, Clintön. ['ö' instead of 'o'] --> match and linked! And the form in the authority record is 'Heylin, Clinton.' So it is now that "punctuation difference" or "whitespace difference" make linker not to link. It is not the current patch that prevents this or changes something here.
I'm a bit curious why you changed the authorities() function in C4/Heading.pm instead of filtering the output of authorities() in C4/Linker/Default.pm, but it looks like that function isn't used anywhere but the linker, so that's probably fine. -- I'm still a bit skeptical about this one, but after reading through everything again... I suppose it's more likely to lead to a lack of links rather than false links... so the risk should be low. That said, if a library has AutoCreateAuthorities enabled, this change could potentially lead to duplicate authorities being created. That said, that's a problematic system preference anyway. -- I still think it would be wise to wrap this in a system preference, so that you don't interfere with existing behaviour. I was tempted to mark Failed QA, but I'll leave it for someone else to decide.
I agree with David here, this needs to be optional. All your points are valid Janusz, but I do think English language catalogers, and possibly others with different sets of diacritics, are used to entering "Beyonce" in a controlled field and having the linker match to "Beyoncé" and automatically having the diacritics added rather than having to enter them on their own. I would have liked to see the matching done in ES directly, but it would require adding an non-analyzed field and matching there, this is a very simple fix, elegant.