Bug 35455

Summary: ICU does not strip = when indexing/searching
Product: Koha Reporter: David Cook <dcook>
Component: Searching - ZebraAssignee: David Cook <dcook>
Status: Pushed to oldstable --- QA Contact: Martin Renvoize <martin.renvoize>
Severity: normal    
Priority: P5 - low CC: fridolin.somers, lucas, martin.renvoize
Version: Main   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
This change fixes an issue with Zebra ICU searching where titles with colons aren't properly searchable, especially when used with Analytics. A full re-index of Zebra is needed for this change to take effect.
Version(s) released in:
24.05.00,23.11.02,23.05.08
Attachments: Bug 35455: Remove = in ICU for indexing/searching
Record 1
Record 2 for testing
Bug 35455: Remove = in ICU for indexing/searching
Bug 35455: Remove = in ICU for indexing/searching

Description David Cook 2023-12-01 01:50:40 UTC
The CHR config file /etc/koha/zebradb/etc/word-phrase-utf.chr replaces = with a space, but the ICU config files /etc/koha/zebradb/etc/words-icu.xml and /etc/koha/zebradb/etc/phrases-icu.xml do not transform the = character.

I encountered this problem because the "Show analytics" link wasn't appearing for a journal record, but it will affect other search contexts too.

Note on https://unicode-org.github.io/icu/userguide/strings/regexp.html it says:

"The property expression \p{punct} differs in what it matches. Java matches matches any of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. From that list, ICU omits $+<=>^`|~     ICU follows the recommendations from Unicode UTS-1"

Arguably we should remove <=>^`|~ during ICU normalization but I'm just focusing on = for now, as the colon : gets changed to = during searches, and subtitles frequently are introduced by : in English.
Comment 1 David Cook 2023-12-01 02:06:01 UTC
For example:

Record 1:
245$a Awesome title: awesome subtitle

Record 2:
245$a Cool article
773$t Awesome title: awesome subtitle

--

At indexing time, this will be indexed as "Awesome title awesome subtitle" as the ICU removes the punctuation. 

At search time, we can't use "Awesome title= awesome subtitle" because the tokens "title" and "title=" won't match. We've got to remove the = since at indexing time it was a :
Comment 2 David Cook 2023-12-01 02:49:02 UTC
Created attachment 159422 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record
Comment 3 David Cook 2023-12-01 02:59:34 UTC
Created attachment 159423 [details]
Record 1

Record 1 for testing
Comment 4 David Cook 2023-12-01 02:59:50 UTC
Created attachment 159424 [details]
Record 2 for testing

Record 2 for testing
Comment 5 David Nind 2023-12-04 02:38:09 UTC
Created attachment 159512 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Signed-off-by: David Nind <david@davidnind.com>
Comment 6 Martin Renvoize 2023-12-19 15:05:57 UTC
Created attachment 160011 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2023-12-19 15:06:18 UTC
All works as described and makes sense.

Passing QA
Comment 8 Katrin Fischer 2023-12-20 14:15:06 UTC
Pushed for 24.05!

Well done everyone, thank you!
Comment 9 Fridolin Somers 2024-01-03 15:33:39 UTC
Pushed to 23.11.x for 23.11.02
Comment 10 Lucas Gass 2024-01-04 21:01:18 UTC
Backported to 23.05.x for upcoming 23.05.08