|
Description
Heather
2020-09-16 15:07:45 UTC
We are using Elastic Search and have noticed a similar problem for authorities with trailing periods. The trailing periods of an authority should be ignored in the sort, but are instead showing at the end of the result list. This problem appears in the advanced editor with the authority look-up (ctrl-shift-L) search. If you were to search for "Cooking." with the search parameter "contains" the result is not returned at the top of the list as expected, but rather towards the bottom. Further complicating, if you use the search parameter "starts with" or "exact" no results are returned. Do the same search without the trailing period "Cooking" and you will get the expected results. Also, Library of Congress is moving toward minimally punctuated authorities in which there is no closing punctuation on some authorities. As a result there are now existing authorities following different punctuation rules. Regardless of the source, trailing periods are throwing off the sort and I also suspect auto linking. Here are some additional papers on punctuation in authorities. https://www.oclc.org/bibformats/en/6xx.html shows examples of the 600 subject tags without punctuation at the end. These are links to a power point and paper on minimally punctuated records put out by Library of Congress. https://www.loc.gov/aba/pcc/sct/documents/GuidelinesMinimallyPunctuatedMARC-SCT-2020-01.pptx https://www.loc.gov/aba/pcc/documents/PCC-Guidelines-Minimally-Punctuated-MARC-Data.docx In any event, Elastic Search appears to need some refinement in how punctuation is handled to bring it into compliance with current practice. Bump, we are experiencing the same problem as described by Esther. I wonder if there could be some similarities to bug 33594 I'm unable to verify it right now, but maybe somebody could -- if adding "alternate: shifted" in admin/searchengine/elasticsearch/field_config.yaml here:
sort:
default:
type: icu_collation_keyword
index: false
numeric: true
+ alternate: shifted
would solve the issue.
(In reply to Katrin Fischer from comment #3) > I wonder if there could be some similarities to bug 33594 I applied the fixes from Bug 33594 but it didn't seem to solve this problem, thank you anyway! (In reply to Janusz Kaczmarek from comment #4) > I'm unable to verify it right now, but maybe somebody could -- if adding > "alternate: shifted" in admin/searchengine/elasticsearch/field_config.yaml > here: > > sort: > default: > type: icu_collation_keyword > index: false > numeric: true > + alternate: shifted > > would solve the issue. This worked at least for upstream, I will attach a patch. Created attachment 150820 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html To test: 1. Create three authority records with the following values: 150 $a Science 150 $a Science $v B 150 $a Science $v C 2. Search for your authority records with "Science" in "Heading A-Z" order The search results will likely be in this order: 1. Science B 2. Science C 3. Science This is an unexpected order 3. Apply the patch and reindex sudo koha-elasticsearch --rebuild -r <instancename> 4. Search for your authority records again with "Science" in "Heading A-Z" order Confirm your search results show in the correct order. 1. Science 2. Science B 3. Science C Sponsored-by: Education Services Australia SCIS I haven't been able to reproduce the problem. I created a sandbox without applying the patch (the sandbox is version 22.12.00.030), and created authority records with these values: 150 _ 0 Science 150 _ 0 Science B 150 _ 0 Science $v Bibliographies 150 _ 0 Science C 150 _ 0 Science $v Congresses (I created all of these because I wasn't sure if the plan meant literally "Science B" in the $a, or create a heading with a subdivision that begins with "B" after the $a Science, so I did both.) In the staff client, searching the $a for: Science Displays these headings sorting correctly: Science Science B Science Bibliographies Science Brazil Periodicals. Science C Science Congresses Likewise an authority search in the OPAC for "starts with" sorted by "heading ascendent" sorts them correctly: Science Science B Science Bibliographies Science Brazil Periodicals. Science C Science Congresses --h2 I added some more headings, with punctuation, since I noticed the problem originally in headings with punctuation. I added topical headings: 150 _ 0 Science (Fictitious character) 150 _ 0 Science Science 150 _ 0 Science-Science 150 _ 0 Science, Science And repeated the above searches, and everything is still sorting correctly without the patch applied, thusly, in both the staff client and the OPAC of the sandbox: Science Science B Science Bibliographies Science Brazil Periodicals. Science C Science Congresses Science fiction. Science (Fictitious character) Science Science Science-Science Science, Science SCIENCES DU SOL However, in my own Koha (version 22.05.11.000), the problem persists, with retrieved authority records sorting: Saint (Fictitious character) Saint Charles, Charles Rather than correctly as: Saint Charles, Charles Saint (Fictitious character) So the problem exists in my Koha catalog, but I can't reproduce the problem in a sandbox, so can't test the patch. I don't know whether to be happy or sad!:) --h2 Created attachment 150899 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html To test: 1. Create three authority records with the following values: 150 $a Science 150 $a Science $v B 150 $a Science $v C 2. Search for your authority records with "Science" in "Heading A-Z" order The search results will likely be in this order: 1. Science B 2. Science C 3. Science This is an unexpected order 3. Apply the patch and reindex sudo koha-elasticsearch --rebuild -r <instancename> 4. Search for your authority records again with "Science" in "Heading A-Z" order Confirm your search results show in the correct order. 1. Science 2. Science B 3. Science C Sponsored-by: Education Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> I've signed this off, but I have a query: 1. If I add the authority records as per step 1 of the test plan, the authority records sort correctly both before and after the patch is applied. 2. If I add some punctuation to the terms, then before applying they don't sort correctly, but after the patch they do. For example: 150 $a Science. (displays last) 150 $a Science $v B (displays second) 150 $a Science $v (C) (displays first) Reading the bug, I'm assuming that the problem we are trying to solve is for terms with punctuation. Testing notes (using koha-testing-docker (KTD)): 1. I tested using Elasticsearch 7 - let me know if it should be tested with other versions, and Open Search. (In reply to Heather from comment #8) > However, in my own Koha (version 22.05.11.000), the problem persists, with > retrieved authority records sorting: > > Saint (Fictitious character) > Saint Charles, Charles > > Rather than correctly as: > Saint Charles, Charles > Saint (Fictitious character) > > So the problem exists in my Koha catalog, but I can't reproduce the problem > in a sandbox, so can't test the patch. I don't know whether to be happy or > sad!:) But did you check "Saint (Fictitious character)" vs. "Saint (Fictitious character)" in the test environment? I must say, with the Aleisha's examples I got always correct results, with or without the patch (there is no punctuation). However with the two Saints I get: Saint (Fictitious character) Saint Charles, Charles without the patch, and: Saint Charles, Charles Saint (Fictitious character) with the patch applied. Aleisha, could you please verify your test data proposal? Additional notes: 1. I added Heather's and Janusz's terms - before the patch is applied they sort incorrectly, after the patch they sort correctly (I can attach some screenshots if that helps). 2. After applying the patch, I restarted everything (flush_memcached and restart_all) and I did a full re-index (koha-elasticsearch --rebuild -d -b -a kohadev). Created attachment 150914 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html To test: 1. Create three authority records with the following values: 150 $a Science. 150 $a Science $v B. 150 $a Science $v C. 2. Search for your authority records with "Science" in "Heading A-Z" order The search results will likely be in this order: 1. Science B. 2. Science C. 3. Science. This is an unexpected order 3. Apply the patch and reindex sudo koha-elasticsearch --rebuild -r <instancename> 4. Search for your authority records again with "Science" in "Heading A-Z" order Confirm your search results show in the correct order. 1. Science. 2. Science B. 3. Science C. Sponsored-by: Education Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> Oops, left the punctuation off my test plan, have amended (In reply to Aleisha Amohia from comment #14) > Oops, left the punctuation off my test plan, have amended Thanks ALeisha. I can confirm that with the change, authorities with punctuation are now sorting correctly (Heading A-Z). Tested using Elasticsearch 7. Have changed the status to Signed Off. David Created attachment 151180 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html To test: 1. Create three authority records with the following values: 150 $a Science. 150 $a Science $v B. 150 $a Science $v C. 2. Search for your authority records with "Science" in "Heading A-Z" order The search results will likely be in this order: 1. Science B. 2. Science C. 3. Science. This is an unexpected order 3. Apply the patch and reindex sudo koha-elasticsearch --rebuild -r <instancename> 4. Search for your authority records again with "Science" in "Heading A-Z" order Confirm your search results show in the correct order. 1. Science. 2. Science B. 3. Science C. Sponsored-by: Education Services Australia SCIS Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net> Works, makes sense, QA script happy, code looks good, passing QA :) Created attachment 151421 [details]
Screenshot of sorting after
Forgive the silly example, but "alternate: shifted" ignores punctuation and whitespace.
Do we want white space considered? i.e. is it correct for 'Santabna' to sort before 'Santa clarita'?
I would expect:
Santa Ana
Santa Clarita
Santa Claus
Santabna
Santana, Carlos
Hi! > Do we want white space considered? i.e. is it correct for 'Santabna' to sort > before 'Santa clarita'? It is not correct. The shorthand rule is "nothing before something." Therefore, "Santabna" should sort after "Santa Clarita." This is correct: > Santa Ana > Santa Clarita > Santa Claus > Santabna > Santana, Carlos The example that illustrates this is in 1.1.1 of the Library of Congress Filing Rules (cited above) (apologies for omited diacritics, but diacritical characters are filed as their non-diacritic English language equivalents): Hand blows Hand book for Prospect Park Hand in glove Handbok for sangere Handbook for adventure Hande im Pflug Hands in the past Handu --h2 Created attachment 151471 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html https://unicode-org.github.io/icu/userguide/collation/concepts.html To test: 1 - Create authorities with main headings like below and confirm they sort in the order shown Hand blows Hand book for Prospect Park Hand (Fictitious character) Hand in glove Hand-ball Handbok for sangere Handbook for adventure Hande im Pflug Hands in the past Handu 2 - Also confirm above order is correct and expected 3 - Test biblio search sorting as well (title, author, etc) I have tried the patch. Maybe I'm doing something wrong but I was unable to get correct results. In fact, applying the patch did not change anything in the order--I'm getting: Hand (Fictitious character) Hand blows Hand book for Prospect Park Hand in glove Hand-ball Handbok for sangere Handbook for adventure Hande im Pflug Hands in the past Handu Same problems with an alternate names set--I'm getting: Kowal L.G. Kowal-Kwiatkowska, Aleksandra Kowal-Orczykowska, Anna Kowal, Alina Kowalak, Alina whereas I would expect: Kowal, Alina Kowal-Kwiatkowska, Aleksandra Kowal L.G. Kowal-Orczykowska, Anna Kowalak, Alina Unfortunately, the original proposal (alternate: shifted) also does not work well here (but the order changes, so there is no question of changing wrong file). Any suggestions? Created attachment 152608 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html https://unicode-org.github.io/icu/userguide/collation/concepts.html To test: 1 - Create authorities with main headings like below and confirm they sort in the order shown Hand blows Hand book for Prospect Park Hand (Fictitious character) Hand in glove Hand-ball Handbok for sangere Handbook for adventure Hande im Pflug Hands in the past Handu 2 - Also confirm above order is correct and expected 3 - Test biblio search sorting as well (title, author, etc) Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com> Created attachment 152646 [details] [review] Bug 26472: Configure Elasticsearch for better alphabetic sorting This enhancement configures the ICU collation keyword plugin used by Elasticsearch for sorting to better handle punctuation and whitespace in sort fields. Details at: https://www.elastic.co/guide/en/elasticsearch/plugins/current/analysis-icu-collation-keyword-field.html https://unicode-org.github.io/icu/userguide/collation/concepts.html To test: 1 - Create authorities with main headings like below and confirm they sort in the order shown Hand blows Hand book for Prospect Park Hand (Fictitious character) Hand in glove Hand-ball Handbok for sangere Handbook for adventure Hande im Pflug Hands in the past Handu 2 - Also confirm above order is correct and expected 3 - Test biblio search sorting as well (title, author, etc) Signed-off-by: Aleisha Amohia <aleishaamohia@hotmail.com> Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov> Signed off! The sort order is correct: Hand-ball Hand blows Hand book Hand (Fictitious character) Etc.! I made a mistake (above) saying that Hand-ball should file, e.g., after Hand in glove. I tested and tested and tested, and the search order with this patch applied is CORRECT!! WOO-HOO!!! ::librarian happy dance!:: (I also verified that search orders are correct for bibs.) I'M SO HAPPY!!! THANK YOU THANK YOU THANK YOU, Most Awesome Community!!! (In reply to Heather from comment #24) > Signed off! The sort order is correct: > > Hand-ball > Hand blows > Hand book > Hand (Fictitious character) Heather, a--maybe--silly question, since you have analyzed this in details: Having a phrase like 'Ha wen' -- should it be placed at the beginning or at the end of the list? Now it is put at the end, whereas my first intuition would be: I want to see it at the beginning (a this is as I would understand "arranged word by word, and words are arranged character by character"). So, am I missing something or we still do not have the desired effect...? I would appreciate your comment. This is not a silly question, Janusz! 'Ha wen' should be at the beginning, like this: Ha wen Hand-ball Hand blows Hand book Hand (Fictitious character) If it is not sorting this way, it is not the desired effect--it is failing & shouldn't be signed off. The patch needs fixing then. I don't have time to re-test before Monday, but I can try again on Monday including the example of 'Ha wen.' Or, if you've already tested with 'Ha wen,' Janusz, and it's not sorting correctly, then it should fail sign off, I think. Thank you for noticing this, Janusz!!! Well done!!!! Hi, again, Janusz! I was not able to reproduce your sorting problem. I created a sandbox, applied the patch, then created authority records for all the headings in the test plan, plus a heading for: Ha wen. All headings sorted correctly: Ha wen Hagahara, Yoshiaki, 1939- Hailstone, H. Haim, Emmanuelle Hall, Joseph N., 1966- Halliwell, Stephen. Hamburger, David Hamel, Debra. Hamilton-James, Charlie. Hamilton-James, Charlie. Hamilton-Jones, Charlie Homes and haunts. Hamilton-Jones, Charlie Homes and haunts. Hamilton, Karen A. Hanʼguk Kiwŏn. Hand-ball Hand blows Hand book for Prospect Park Hand (Fictitious character) Hand in glove Handbok for sangere Handbook for adventure Hande im Pflug Handel, George Frideric, 1685-1759. Handel, George Frideric, 1685-1759. Alcina. Tornami a vagheggiar. Handel, George Frideric, 1685-1759. Amadigi. Ah! spietato! e non ti muove. Handel, George Frideric, 1685-1759. Apollo e Dafne. Felicissima quest'alma. Handel, George Frideric, 1685-1759. Ariodante. Mio crudel martoro. Handel, George Frideric, 1685-1759 Film and video adaptations. Handel, George Frideric, 1685-1759 Giulio Cesare. Handel, George Frideric, 1685-1759. Giulio Cesare. Da tempeste il legno infranto. Handel, George Frideric, 1685-1759. Giulio Cesare. Piangerò la sorte mia. Handel, George Frideric, 1685-1759. Rinaldo. Lascia ch'io pianga mia cruda sorte. Handel, George Frideric, 1685-1759. Rinaldo. Vo' far guerra. Handel, George Frideric, 1685-1759. Semele. Endless pleasure, endless love (Air) Handel, George Frideric, 1685-1759. Semele. Myself I shall adore. Handel, George Frideric, 1685-1759. Teseo. Dolce riposo, ed innocente pace. Handel, George Frideric, 1685-1759. Teseo. O stringerò nel sen. Hands in the past Handu Harmes, Ross. I even created two authority records for "Ha wen," one as a topical term and another as a personal name (in case the type of heading was a problem), and both authority records for "Ha wen" sorted correctly. Based on this, I think the bug should remain signed off--unless Janusz's problem can be re-created? Hi Heater! Thank you very much for checking it. I can confirm--with 'Ha wen' things look good now. I'm working with ktd--I had to restart plack to get the order changed. But still there are doubts. Could you, please, have a look at these three names: Haendel Gregory Haendel-Adler, John Haendel, Georg (mind the presence or absence of the comma and the dash) I get them in the above order whereas I would wish: Haendel-Adler, John Haendel, Georg Haendel Gregory So, I am confused... Am I doing something wrong? Could you, please, check it again? Hi Janusz,
This is the correct order:
>
> Haendel Gregory
> Haendel-Adler, John
> Haendel, Georg
>
The policy is 'nothing' before 'something' so spaces will be sorted before punctuation.
Hope that helps!
Aleisha
Hi Aleisha, Thank you very much for you comment. I'm not sure in what sense this order is correct. If it means that the order conforms to what ICU sort with "strength: quaternary" should return, then OK. But I am still not sure if it conforms to the LCC Filling Rules (especially p. 1.3.1) cited by Heated in the initial report. Let's take the initial "Saint" example. I am getting (with the patch): Saint (Fictitious character) Saint Charles School Saint Hilaire Institute Saint-Nazaire (France) Saint, Eva Marie Do you get the same order? At the same time--according to Heated, and also to my intuition--it should be: Saint Charles School Saint, Eva Marie Saint (Fictitious character) Saint Hilaire Institute Saint-Nazaire (France) So, I am not sure if--assuming that I understand it well and correctly applied the patch--we get the expected result... If the current patches aren't working as needed, we may want to look into building the authority sort field using: https://metacpan.org/pod/MARC::Field::Normalize::NACO Or something similar Hi, Aleisha-- I don't think so. In https://babel.hathitrust.org/cgi/pt?id=mdp.39015022080140&view=1up&seq=63 Rule 12 states, "Words connected by a hyphen are always treated as separate words. This rule applies even when the first part of a hyphenated word is a prefix that sometimes appears as an integral part of the word. This rule also applies to compound surnames." So, yes, "nothing before something," but a hyphen is treated as "nothing," i.e., a space. So I agree with Janusz that "Haendel-Adler, John" should be first in the example. Most insitutions agree, too, that the order should then be: Haendel-Adler, John Haendel, Georg Haendel Gregory The Library of Congress Filing Rules go deeply into intricacies...but Koha is a global community. In this case, though, I think LC Filing Rules are agreeing with most institutions worldwide that the order should be: Haendel-Adler, John Haendel, Georg Haendel Gregory If we go more deeply into the LC Filing Rules, we could come up with examples that are different based on whether the heading is a surname, a topic, a place, etc., and which elements of the field we're filing on, but the Rules were written at a time when the vast majority of libraries had humans filing cards in different card catalogs: an author/title catalog, and a different subject catalog. We are now in an era when keyword searching returns results from multiple indexes, and this makes the most sense and agrees with the basic tenets of common filing rules: Haendel-Adler, John Haendel, Georg Haendel Gregory Aleisha--shall I test the patch again? Nick: I think we're really close! Cheerio!!! h2 (In reply to Nick Clemens from comment #18) > Forgive the silly example, but "alternate: shifted" ignores punctuation and > whitespace. > > Do we want white space considered? i.e. is it correct for 'Santabna' to sort > before 'Santa clarita'? Thanks for catching this! :o --- (In reply to Heather from comment #32) > I don't think so. In > https://babel.hathitrust.org/cgi/pt?id=mdp.39015022080140&view=1up&seq=63 > Rule 12 states, "Words connected by a hyphen [...] So if I understand correctly. The current proposal should sort " " (space) after "-","," and other symbols? How much of a problem is it? The 1st problematic cases on this ticket caused the results to grouped by punctuation. Which was bad because it was randomly mixing stuff that semantically was linked and should have been together, the letters have the real meaning here. Not a hyphen. comment 18 also shows stuff mixed that splat the group of "santa " that should have stayed all together. So about sorting " " after "-" and "," => does it lead to examples of stuff that is in the wrong place as bad as the two above cases? Or is the rule more about having a standardized unambiguous order for the sake of consistency and predictability even if there was nothing semantically messed up without that rule. splat => split (past participle) Hi! I think it would be an OK fix if it sorts " "(space) after hyphens and other punctuation, because that is clear to the user. It may not comply exactly to Library of Congress Filing Rules, but Koha is for the whole world and not everyone wants to comply with Library of Congress Filing Rules! A fix is NEEDED to make it better, and this would be better. I think it is for a developer to say if it is easier and quicker to make this patch comply to Library of Congress Filing Rules (i.e., treat hyphens, commas, etc. as spaces) NOW and delay the fix, or just get this fix out there to make it better now and create another bug to let the library decide if they want spaces sorted after hyphens & other punctuation, or to have hyphens & commas & such treated as spaces in the sorting (via a new syspref?). --h2 > It may not comply exactly to Library of Congress Filing Rules, but Koha is for the whole world and not everyone wants to comply with Library of Congress Filing Rules! That wouldn't prevent someone to submit a change in another ticket to implement the Library of Congress Filing Rules. From a functional POV if there is no other convention that this would conflict with, then there wouldn't be an issue to change that last sorting rule. > I think it is for a developer to say if it is easier and quicker to make this patch comply to Library of Congress Filing Rules (i.e., treat hyphens, commas, etc. as spaces) About that, I'm not sure there is a config in ICU or Elastic that changes only what we need without messing up all the rest. For reference if someone wants to dig into that: https://www.elastic.co/guide/en/elasticsearch/plugins/7.17/analysis-icu-collation-keyword-field.html https://unicode-org.github.io/icu-docs/apidoc/released/icu4j/com/ibm/icu/text/Collator.html > or just get this fix out there to make it better now We might be in this situation. It seems getting the last detail (which isn't strictly the initial report but a thread that lead us to a clew of yarn ^^) suddenly greatly raises the time to make a working solution. Other than that, the current proposal has no known potential issue, right? > 1 - Create authorities with main headings like below and confirm they sort in the order shown
> Hand blows
> Hand book for Prospect Park
> Hand (Fictitious character)
> Hand in glove
> Hand-ball
> Handbok for sangere
> Handbook for adventure
> Hande im Pflug
> Hands in the past
> Handu
nope T_T
i get this:
Hand (Fictitious character)
Hand blows
Hand book for Prospect Park
Hand in glove
Hand-ball
Handbok for sangere
Handbook for adventure
Hande im Pflug
Hands in the past
Handu
With ES 7 and opensearch 1.x
And I double checked not using zebra instead of ES.
My koha (ktd) was started with the patch already applied.
Any idea of what could go wrong? Any implicit step before the 1st in the test plan ?
Another question: how does this change work for existing instances?
I had a koha on main/master.
- applied the patch
- restarted the ES container (waited for it to finish)
- restarted services
- reindexed (because the container didn't have persistence)
misc/search_tools/rebuild_elasticsearch.pl -v -d
And the search still had the same order :o
---
> 3 - Test biblio search sorting as well (title, author, etc)
Is it about just making sure the catalogue search and the an:XX syntax give sane results or something more specific?
(In reply to Victor Grousset/tuxayo from comment #38) > Another question: how does this change work for existing instances? > I had a koha on main/master. > - applied the patch > - restarted the ES container (waited for it to finish) > - restarted services > - reindexed (because the container didn't have persistence) > misc/search_tools/rebuild_elasticsearch.pl -v -d > > > And the search still had the same order :o > > > --- > > > > 3 - Test biblio search sorting as well (title, author, etc) > > Is it about just making sure the catalogue search and the an:XX syntax give > sane results or something more specific? Try -r instead of -d and confirm you are using ktd and/or confirm your koha-conf is pointing to the updated file form this patch: 362 <!-- Uncomment the following line if you want to override the Elasticsearch default field settings --> 363 <!-- <elasticsearch_field_config>/etc/koha/sites/kohadev/searchengine/elasticsearch/field_config.yaml</elasticsea rch_field_config> --> > confirm your koha-conf is pointing to the updated file form this patch: It's as expected commented. So then it would use the default (updating by this patch): $config_file ||= C4::Context->config('intranetdir') . '/admin/searchengine/elasticsearch/index_config.yaml'; > confirm you are using ktd yes, even more «My koha (ktd) was started with the patch already applied.» So in any case, the field_config.yaml is the right and no need to reindex or reset the mappings (thanks for the -r, I'll keep that in mind) Retried and still got a wrong order, hmmm [U+1F614] (In reply to Victor Grousset/tuxayo from comment #37) > > 1 - Create authorities with main headings like below and confirm they sort in the order shown > > Hand blows > > Hand book for Prospect Park > > Hand (Fictitious character) > > Hand in glove > > Hand-ball > > Handbok for sangere > > Handbook for adventure > > Hande im Pflug > > Hands in the past > > Handu > > nope T_T > > i get this: > Hand (Fictitious character) > Hand blows > Hand book for Prospect Park > Hand in glove > Hand-ball > Handbok for sangere > Handbook for adventure > Hande im Pflug > Hands in the past > Handu > > With ES 7 and opensearch 1.x > And I double checked not using zebra instead of ES. > My koha (ktd) was started with the patch already applied. > > Any idea of what could go wrong? Any implicit step before the 1st in the > test plan ? Restesting, this is what I get, and rereading, this is what is expected with quaternary - it considers differences at the 4th level, meaning accents and punctuation are both considered So "Hand (Fictitious character)" comes first because the space and then the '(' come before the letters It makes other things sort too: hand book for Prospect Park Hand book for Prospect Park Hand boôk for Prospect Park Hand böok for Prospect Park It seems what we want is "punctuation ignored, but whitespace not ignored" but I don't seem to find that option: https://unicode-org.github.io/icu/userguide/collation/architecture.html#strength-level https://unicode-org.github.io/icu/userguide/collation/customization/ignorepunct.html > Restesting, this is what I get, and rereading, this is what is expected with quaternary - it considers differences at the 4th level, meaning accents and punctuation are both considered
oops so the patch doesn't match the specs/test plan.
So "strength: quaternary" is still better than "alternate: shifted", right? And better than the current sorting.
Not sure there is another low cost alternative from all the previous comments.
> It seems what we want is "punctuation ignored, but whitespace not ignored" but I don't seem to find that option:
So in ICU itself there isn't that option, it's not about the analysis-icu plugin that doesn't expose them?
Asked in the ICU support mailing list, let's see if we get something :)
https://sourceforge.net/p/icu/mailman/icu-support/thread/d1a3c944-02a7-4cc0-87ad-4ad30b773967%40tuxayo.net/#msg37894734 Someone said: > I think you might be able to get that by reordering spaces above punctuation, and setting the first non ignorable to be the first space. Does anyone know if we can do such a thing? ^^" https://sourceforge.net/p/icu/mailman/icu-support/thread/d1a3c944-02a7-4cc0-87ad-4ad30b773967%40tuxayo.net/#msg37895241 > > I think you might be able to get that by reordering spaces above > > punctuation, and setting the first non ignorable to be the first space. > > > > No, I am pretty sure that alternate=shifted compares the primary weight > with the maxVariable setting before script reordering. > > I just tried this in the ICU Collation Demo > <https://icu4c-demos.unicode.org/icu-bin/collation.html> with > > - rule "[reorder punct space]" > - alternate=shifted > - max variable=punct > > and both spaces and punctuation get shifted/ignored. > > So I don't think we have a way to ignore/shift anything other than > "anything up to the max variable". > > You might be best off pre-processing the strings, removing punctuation > characters before sending the string into collation. Quite over my head but it seems to confirm no way to get what we want with ICU config. If the assumptions are correct - punctuation ignored, but whitespace not ignored is our need - strength: quaternary" is still better than "alternate: shifted" and better than the current sorting - might need more testing since it managed to masquerade as complying with the test plan ^^" Then immediate move forward is go on with the "strength: quaternary" patch. (In reply to Victor Grousset/tuxayo from comment #45) > > If the assumptions are correct > - punctuation ignored, but whitespace not ignored is our need > - strength: quaternary" is still better than "alternate: shifted" and better > than the current sorting > - might need more testing since it managed to masquerade as complying with > the test plan ^^" > > Then immediate move forward is go on with the "strength: quaternary" patch. In that case, I will set this patch to Needs signoff. Failing QA for not matching the test plan, then. If strength: quaternary is actually in some way better than what we have now, we need a test plan that shows that, rather than one which lists what we really want to get but do not get from the patch :) (In reply to Aleisha Amohia from comment #46) > In that case, I will set this patch to Needs signoff. Great, thanks, let's move on that way :D (In reply to Phil Ringnalda from comment #47) > If strength: quaternary is actually in some way better than what we have > now, we need a test plan that shows that, rather than one which lists what > we really want to get but do not get from the patch :) indeed, +1 Anyone welcome to propose an updated test plan and I would be happy to test. And replace the existing on the patch in case that's tedious or someone isn't versed into juggling with git and bugzilla. Created attachment 173268 [details]
Sample auths for testing
I played with something like this here: https://stackoverflow.com/questions/75646849/how-to-write-a-collation-rule-for-elasticsearch-icu-collation-keyword-field-wit Editing the field_config to add: rules: "[reorder space punct Latn digit currency others]" alternate: non-ignorable Moving around the things in rules you can get pretty close, still not 100% |