Bug 35455 - ICU does not strip = when indexing/searching
Summary: ICU does not strip = when indexing/searching
Status: Pushed to oldstable
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: David Cook
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2023-12-01 01:50 UTC by David Cook
Modified: 2024-01-04 21:01 UTC (History)
3 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This change fixes an issue with Zebra ICU searching where titles with colons aren't properly searchable, especially when used with Analytics. A full re-index of Zebra is needed for this change to take effect.
Version(s) released in:
24.05.00,23.11.02,23.05.08


Attachments
Bug 35455: Remove = in ICU for indexing/searching (2.40 KB, patch)
2023-12-01 02:49 UTC, David Cook
Details | Diff | Splinter Review
Record 1 (1.00 KB, application/xml)
2023-12-01 02:59 UTC, David Cook
Details
Record 2 for testing (1.10 KB, application/xml)
2023-12-01 02:59 UTC, David Cook
Details
Bug 35455: Remove = in ICU for indexing/searching (2.45 KB, patch)
2023-12-04 02:38 UTC, David Nind
Details | Diff | Splinter Review
Bug 35455: Remove = in ICU for indexing/searching (2.52 KB, patch)
2023-12-19 15:05 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description David Cook 2023-12-01 01:50:40 UTC
The CHR config file /etc/koha/zebradb/etc/word-phrase-utf.chr replaces = with a space, but the ICU config files /etc/koha/zebradb/etc/words-icu.xml and /etc/koha/zebradb/etc/phrases-icu.xml do not transform the = character.

I encountered this problem because the "Show analytics" link wasn't appearing for a journal record, but it will affect other search contexts too.

Note on https://unicode-org.github.io/icu/userguide/strings/regexp.html it says:

"The property expression \p{punct} differs in what it matches. Java matches matches any of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. From that list, ICU omits $+<=>^`|~     ICU follows the recommendations from Unicode UTS-1"

Arguably we should remove <=>^`|~ during ICU normalization but I'm just focusing on = for now, as the colon : gets changed to = during searches, and subtitles frequently are introduced by : in English.
Comment 1 David Cook 2023-12-01 02:06:01 UTC
For example:

Record 1:
245$a Awesome title: awesome subtitle

Record 2:
245$a Cool article
773$t Awesome title: awesome subtitle

--

At indexing time, this will be indexed as "Awesome title awesome subtitle" as the ICU removes the punctuation. 

At search time, we can't use "Awesome title= awesome subtitle" because the tokens "title" and "title=" won't match. We've got to remove the = since at indexing time it was a :
Comment 2 David Cook 2023-12-01 02:49:02 UTC
Created attachment 159422 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record
Comment 3 David Cook 2023-12-01 02:59:34 UTC
Created attachment 159423 [details]
Record 1

Record 1 for testing
Comment 4 David Cook 2023-12-01 02:59:50 UTC
Created attachment 159424 [details]
Record 2 for testing

Record 2 for testing
Comment 5 David Nind 2023-12-04 02:38:09 UTC
Created attachment 159512 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Signed-off-by: David Nind <david@davidnind.com>
Comment 6 Martin Renvoize 2023-12-19 15:05:57 UTC
Created attachment 160011 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 7 Martin Renvoize 2023-12-19 15:06:18 UTC
All works as described and makes sense.

Passing QA
Comment 8 Katrin Fischer 2023-12-20 14:15:06 UTC
Pushed for 24.05!

Well done everyone, thank you!
Comment 9 Fridolin Somers 2024-01-03 15:33:39 UTC
Pushed to 23.11.x for 23.11.02
Comment 10 Lucas Gass 2024-01-04 21:01:18 UTC
Backported to 23.05.x for upcoming 23.05.08