Bug 6098 - zebra conf NSB NSE indexed as space
Summary: zebra conf NSB NSE indexed as space
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: 3.6
Hardware: All All
: P3 normal (vote)
Assignee: Fridolin Somers
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks: 5885
  Show dependency treegraph
 
Reported: 2011-04-06 15:43 UTC by Fridolin Somers
Modified: 2013-12-05 19:57 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
pro (1.46 KB, patch)
2011-04-06 15:46 UTC, Fridolin Somers
Details | Diff | Splinter Review
Clean Output.pm (1.59 KB, patch)
2011-05-13 14:15 UTC, Frédéric Demians
Details | Diff | Splinter Review
BUG 6098 zebra conf NSB NSE indexed as space (1.51 KB, patch)
2011-12-21 19:16 UTC, Chris Cormack
Details | Diff | Splinter Review
Bug 6098 Followup -- doesn't replace NSB/NSE by space (1.63 KB, patch)
2011-12-21 19:16 UTC, Chris Cormack
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2011-04-06 15:43:48 UTC
For those who are not using icu in Zebra conf and have NSB NSE special characters in biblio.
This patch removes those undisplayed characters in searching and sorting.

Regards,
Comment 1 Fridolin Somers 2011-04-06 15:46:14 UTC Comment hidden (obsolete)
Comment 2 Frédéric Demians 2011-05-03 13:25:05 UTC
Isn't this bug already solved in current
/etc/zebradb/etc/word-phrase-utf.chr file.

We have at the end of the file:

  map (\x88.*\x89))   @
Comment 3 Fridolin Somers 2011-05-03 14:42:10 UTC
(In reply to comment #2)
> Isn't this bug already solved in current
> /etc/zebradb/etc/word-phrase-utf.chr file.
> 
> We have at the end of the file:
> 
>   map (\x88.*\x89))   @

Indeed,

This line should remove from indexation all characters (regexp ".*") between "\x88" and "\x89".

Unfortunately, I have tested a lot, is doesn't work.

It seems that Zebra understands in "map" only a few elements of regexp syntaxe like "\s". It is impossible tu define a unknown number of characters.

Maybe it is limited to avoid a catastrophic indexing performance.
Comment 4 Frédéric Demians 2011-05-13 14:14:39 UTC
On any page, NSB/NSE characters are replaced by space in Output.pm. It
isn't a good thing:
  
  - It isn't necessary since Zebra indexer handle properly those
    characters as delimiter.
  - It isn't necessary for display since XSL file already do that.
  - It isn't efficient since a regex is done on every whole page sent by
    Koha, not only on biblio record part.
  - It removes an information that librarian want to keep in their
    biblio records: any record edited in Koha loose its NSB/NSE
    characters.

See proposed patch.
Comment 5 Frédéric Demians 2011-05-13 14:15:11 UTC Comment hidden (obsolete)
Comment 6 Fridolin Somers 2011-05-13 16:24:57 UTC
(In reply to comment #5)
> Created attachment 4105 [details] [review]
> Clean Output.pm

Shoudn't it be 2 different bugs ?
Comment 7 Frédéric Demians 2011-05-13 16:36:10 UTC
> Shoudn't it be 2 different bugs ?

Maybe. IMHO, we should just have one bug labeled "NSB/NSE proper
handling". This bug stays open until we consider there isn't anything to
do anymore to handle correctly those characters in various places. And
after closing this bug, we can reopen it if we discover another corner
where NSB/NSE have to be eradicated!

So open a new bug if you think it's better, or rename this one (not sur
it's possible), or/and sign-of my patch if you agree that it's required,
or argue against it. Just for performace reason, I think it's very
necessary.
Comment 8 Chris Cormack 2011-08-01 08:36:48 UTC
Second patch requires sign off still
Comment 9 Chris Cormack 2011-12-21 19:16:21 UTC
Created attachment 6904 [details] [review]
BUG 6098 zebra conf NSB NSE indexed as space

Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 10 Chris Cormack 2011-12-21 19:16:57 UTC
Created attachment 6905 [details] [review]
Bug 6098 Followup -- doesn't replace NSB/NSE by space

On any page, NSB/NSE characters are replaced by space in Output.pm. It isn't a
good thing:

  - It isn't necessary since Zebra indexer handle properly those characters as
    delimiter.
  - It isn't necessary for display since XSL file already do that.
  - It isn't efficient since a regex is done on every whole page sent by Koha,
    not only on biblio record part.
  - It removes an information that librarian want to keep in their biblio
    records: any record edited in Koha loose its NSB/NSE characters.

Signed-off-by: Frédéric Demians <f.demians@tamil.fr>
Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Comment 11 Ian Walls 2012-01-09 12:49:12 UTC
Simple changes to handling of NSB and NSE characters; removes regex processing and adds to Zebra config.  Marking as Passed QA
Comment 12 Paul Poulain 2012-01-13 10:24:26 UTC
patch pushed, please test

QA COMMENT:

seing comments 2 and 3, shouldn't the line
  map (\x88.*\x89))   @

also be removed from /etc/zebradb/etc/word-phrase-utf.chr file ?