Bug 13064 - Indexing problem with ICU on control characters
Summary: Indexing problem with ICU on control characters
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Fridolin Somers
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-10-10 13:02 UTC by Fridolin Somers
Modified: 2015-06-04 23:33 UTC (History)
1 user (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 13064 - Indexing problem with ICU on control characters (2.34 KB, patch)
2014-10-10 13:13 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 13064 - Indexing problem with ICU on control characters (2.40 KB, patch)
2014-10-14 12:28 UTC, Chris Cormack
Details | Diff | Splinter Review
[PASSED QA] Bug 13064 - Indexing problem with ICU on control characters (2.47 KB, patch)
2014-10-24 14:13 UTC, Kyle M Hall
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2014-10-10 13:02:53 UTC
The ICU configuration files contains a rule to remove control characters :
  <transform rule="[:Control:] Any-Remove"/>
This rule is before tokenization.

The problem is that "[:Control:]" regex contains linefeed, carriage return and tab. See http://www.regular-expressions.info/posixbrackets.html.
So when several lines are indexed, last word of line is joined with first line of next line. Thoses words are then not searchable.

For example :
  First line
  Second line
This will become "First lineSecond line", tokenized as "First", "lineSecond" and "line".
Comment 1 Fridolin Somers 2014-10-10 13:13:34 UTC Comment hidden (obsolete)
Comment 2 Chris Cormack 2014-10-14 12:28:28 UTC Comment hidden (obsolete)
Comment 3 Kyle M Hall 2014-10-24 14:13:51 UTC
Created attachment 32676 [details] [review]
[PASSED QA] Bug 13064 - Indexing problem with ICU on control characters

The ICU configuration files contains a rule to remove control characters :
  <transform rule="[:Control:] Any-Remove"/>
This rule is before tokenization.

The problem is that "[:Control:]" regex contains line feed, carriage return and tab. See http://www.regular-expressions.info/posixbrackets.html.
So when several lines are indexed, last word of line is joined with first line of next line. Thoses words are then not searchable.

For example :
  First line
  Second line
This will become "First lineSecond line", tokenized as "First", "lineSecond" and "line".

Test plan :
- Use ICU in Zebra configuration
- Choose an indexed field, like 300$a
- Create a new record
- Enter several lines in choosen field, like :
  First line
  Second line
- Index this record
=> Without patch the search on "Second" does not return the record
=> With patch the search on "Second" returns the record
- Same tests with tab and carriage return instead of line feed

Signed-off-by: Chris Cormack <chris@bigballofwax.co.nz>

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 4 Tomás Cohen Arazi 2014-11-14 16:17:32 UTC
Patch pushed to master.

Thanks Fridolin!