The CHR config file /etc/koha/zebradb/etc/word-phrase-utf.chr replaces = with a space, but the ICU config files /etc/koha/zebradb/etc/words-icu.xml and /etc/koha/zebradb/etc/phrases-icu.xml do not transform the = character. I encountered this problem because the "Show analytics" link wasn't appearing for a journal record, but it will affect other search contexts too. Note on https://unicode-org.github.io/icu/userguide/strings/regexp.html it says: "The property expression \p{punct} differs in what it matches. Java matches matches any of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. From that list, ICU omits $+<=>^`|~ ICU follows the recommendations from Unicode UTS-1" Arguably we should remove <=>^`|~ during ICU normalization but I'm just focusing on = for now, as the colon : gets changed to = during searches, and subtitles frequently are introduced by : in English.
For example: Record 1: 245$a Awesome title: awesome subtitle Record 2: 245$a Cool article 773$t Awesome title: awesome subtitle -- At indexing time, this will be indexed as "Awesome title awesome subtitle" as the ICU removes the punctuation. At search time, we can't use "Awesome title= awesome subtitle" because the tokens "title" and "title=" won't match. We've got to remove the = since at indexing time it was a :
Created attachment 159422 [details] [review] Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record
Created attachment 159423 [details] Record 1 Record 1 for testing
Created attachment 159424 [details] Record 2 for testing Record 2 for testing
Created attachment 159512 [details] [review] Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record Signed-off-by: David Nind <david@davidnind.com>
Created attachment 160011 [details] [review] Bug 35455: Remove = in ICU for indexing/searching This change more closely aligns ICU and CHR so that ICU also removes the = character. This fixes issues in ICU when searching with a : which gets transformed into a =. Without this change, the Analytics features won't work for titles with a colon in them. Test plan: 0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools) 1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml 2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml 3. vi /etc/koha/zebradb/etc/default.idx Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w" and "icuchain phrases-icu.xml" for "index p" 4. koha-zebra --stop kohadev 5. pkill zebrasrv 6. koha-zebra --start kohadev 7. koha-rebuild-zebra -a -b -f -v kohadev 8. Search for "Awesome title" and open the detail page 9. Note that the "Analytics: Show analytics" line shows up 10. Click that link 11. Note that it opens the "Cool article" record and it displays "In: Awesome title: awesome subtitle" 12. Click that link 13. Note that it opens the "Awesome title" record Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
All works as described and makes sense. Passing QA
Pushed for 24.05! Well done everyone, thank you!
Pushed to 23.11.x for 23.11.02
Backported to 23.05.x for upcoming 23.05.08