Bug 35455

Summary:	ICU does not strip = when indexing/searching
Product:	Koha	Reporter:	David Cook <dcook>
Component:	Searching - Zebra	Assignee:	David Cook <dcook>
Status:	Pushed to oldstable ---	QA Contact:	Martin Renvoize (ashimema) <martin.renvoize>
Severity:	normal
Priority:	P5 - low	CC:	fridolin.somers, lucas
Version:	Main
Hardware:	All
OS:	All
Change sponsored?:	---	Patch complexity:	---
Documentation contact:		Documentation submission:
Text to go in the release notes:	This change fixes an issue with Zebra ICU searching where titles with colons aren't properly searchable, especially when used with Analytics. A full re-index of Zebra is needed for this change to take effect.	Version(s) released in:	24.05.00,23.11.02,23.05.08
Circulation function:
Attachments:	Bug 35455: Remove = in ICU for indexing/searching Record 1 Record 2 for testing Bug 35455: Remove = in ICU for indexing/searching Bug 35455: Remove = in ICU for indexing/searching

Description David Cook 2023-12-01 01:50:40 UTC

The CHR config file /etc/koha/zebradb/etc/word-phrase-utf.chr replaces = with a space, but the ICU config files /etc/koha/zebradb/etc/words-icu.xml and /etc/koha/zebradb/etc/phrases-icu.xml do not transform the = character.

I encountered this problem because the "Show analytics" link wasn't appearing for a journal record, but it will affect other search contexts too.

Note on https://unicode-org.github.io/icu/userguide/strings/regexp.html it says:

"The property expression \p{punct} differs in what it matches. Java matches matches any of !"#$%&'()*+,-./:;<=>?@[\]^_`{|}~. From that list, ICU omits $+<=>^`|~     ICU follows the recommendations from Unicode UTS-1"

Arguably we should remove <=>^`|~ during ICU normalization but I'm just focusing on = for now, as the colon : gets changed to = during searches, and subtitles frequently are introduced by : in English.

Comment 1 David Cook 2023-12-01 02:06:01 UTC

For example:

Record 1:
245$a Awesome title: awesome subtitle

Record 2:
245$a Cool article
773$t Awesome title: awesome subtitle

--

At indexing time, this will be indexed as "Awesome title awesome subtitle" as the ICU removes the punctuation. 

At search time, we can't use "Awesome title= awesome subtitle" because the tokens "title" and "title=" won't match. We've got to remove the = since at indexing time it was a :

Comment 2 David Cook 2023-12-01 02:49:02 UTC

Created attachment 159422 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Comment 3 David Cook 2023-12-01 02:59:34 UTC

Created attachment 159423 [details]
Record 1

Record 1 for testing

Comment 4 David Cook 2023-12-01 02:59:50 UTC

Created attachment 159424 [details]
Record 2 for testing

Record 2 for testing

Comment 5 David Nind 2023-12-04 02:38:09 UTC

Created attachment 159512 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Signed-off-by: David Nind <david@davidnind.com>

Comment 6 Martin Renvoize (ashimema) 2023-12-19 15:05:57 UTC

Created attachment 160011 [details] [review]
Bug 35455: Remove = in ICU for indexing/searching

This change more closely aligns ICU and CHR so that ICU also
removes the = character. This fixes issues in ICU when searching
with a : which gets transformed into a =. Without this change,
the Analytics features won't work for titles with a colon in them.

Test plan:
0. Apply the patch and import bibs from Bugzilla (using Staged MARC tools)
1. cp ./etc/zebradb/etc/phrases-icu.xml /etc/koha/zebradb/etc/phrases-icu.xml
2. cp ./etc/zebradb/etc/words-icu.xml /etc/koha/zebradb/etc/words-icu.xml
3. vi /etc/koha/zebradb/etc/default.idx
Change "charmap word-phrase-utf.chr" to "icuchain words-icu.xml" for "index w"
and "icuchain phrases-icu.xml" for "index p"
4. koha-zebra --stop kohadev
5. pkill zebrasrv
6. koha-zebra --start kohadev
7. koha-rebuild-zebra -a -b -f -v kohadev

8. Search for "Awesome title" and open the detail page
9. Note that the "Analytics: Show analytics" line shows up
10. Click that link
11. Note that it opens the "Cool article" record and it displays
"In: Awesome title: awesome subtitle"
12. Click that link
13. Note that it opens the "Awesome title" record

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>

Comment 7 Martin Renvoize (ashimema) 2023-12-19 15:06:18 UTC

All works as described and makes sense.

Passing QA

Comment 8 Katrin Fischer 2023-12-20 14:15:06 UTC

Pushed for 24.05!

Well done everyone, thank you!

Comment 9 Fridolin Somers 2024-01-03 15:33:39 UTC

Pushed to 23.11.x for 23.11.02

Comment 10 Lucas Gass (lukeg) 2024-01-04 21:01:18 UTC

Backported to 23.05.x for upcoming 23.05.08