The CHR config file /etc/koha/zebradb/etc/word-phrase-utf.chr replaces = with a space, but the ICU config files /etc/koha/zebradb/etc/words-icu.xml and /etc/koha/zebradb/etc/phrases-icu.xml do not transform the = character. I encountered this problem because the "Show analytics" link wasn't appearing for a journal record, but it will affect other search contexts too. Note on https://unicode-org.github.io/icu/userguide/strings/regexp.html it says: "The property expression \p{punct} differs in what it matches. Java matches matches any of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. From that list, ICU omits $+<=>^`|~ ICU follows the recommendations from Unicode UTS-1" Arguably we should remove <=>^`|~ during ICU normalization but I'm just focusing on = for now, as the colon : gets changed to = during searches, and subtitles frequently are introduced by : in English.
For example: Record 1: 245$a Awesome title: awesome subtitle Record 2: 245$a Cool article 773$t Awesome title: awesome subtitle -- At indexing time, this will be indexed as "Awesome title awesome subtitle" as the ICU removes the punctuation. At search time, we can't use "Awesome title= awesome subtitle" because the tokens "title" and "title=" won't match. We've got to remove the = since at indexing time it was a :
Created attachment 159422 [details] [review] Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record
Created attachment 159423 [details] Record 1 Record 1 for testing
Created attachment 159424 [details] Record 2 for testing Record 2 for testing
Created attachment 159512 [details] [review] Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record Signed-off-by: David Nind <david@davidnind.com>
Created attachment 160011 [details] [review] Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
All works as described and makes sense. Passing QA
Pushed for 24.05! Well done everyone, thank you!
Pushed to 23.11.x for 23.11.02
Backported to 23.05.x for upcoming 23.05.08
Not backporting to 22.11