For those who are not using icu in Zebra conf and have NSB NSE special characters in biblio. This patch removes those undisplayed characters in searching and sorting. Regards,
Created attachment 3758 [details] [review] pro
Isn't this bug already solved in current /etc/zebradb/etc/word-phrase-utf.chr file. We have at the end of the file: map (\x88.*\x89)) @
(In reply to comment #2) > Isn't this bug already solved in current > /etc/zebradb/etc/word-phrase-utf.chr file. > > We have at the end of the file: > > map (\x88.*\x89)) @ Indeed, This line should remove from indexation all characters (regexp ".*") between "\x88" and "\x89". Unfortunately, I have tested a lot, is doesn't work. It seems that Zebra understands in "map" only a few elements of regexp syntaxe like "\s". It is impossible tu define a unknown number of characters. Maybe it is limited to avoid a catastrophic indexing performance.
On any page, NSB/NSE characters are replaced by space in Output.pm. It isn't a good thing: - It isn't necessary since Zebra indexer handle properly those characters as delimiter. - It isn't necessary for display since XSL file already do that. - It isn't efficient since a regex is done on every whole page sent by Koha, not only on biblio record part. - It removes an information that librarian want to keep in their biblio records: any record edited in Koha loose its NSB/NSE characters. See proposed patch.
Created attachment 4105 [details] [review] Clean Output.pm
(In reply to comment #5) > Created attachment 4105 [details] [review] > Clean Output.pm Shoudn't it be 2 different bugs ?
> Shoudn't it be 2 different bugs ? Maybe. IMHO, we should just have one bug labeled "NSB/NSE proper handling". This bug stays open until we consider there isn't anything to do anymore to handle correctly those characters in various places. And after closing this bug, we can reopen it if we discover another corner where NSB/NSE have to be eradicated! So open a new bug if you think it's better, or rename this one (not sur it's possible), or/and sign-of my patch if you agree that it's required, or argue against it. Just for performace reason, I think it's very necessary.
Second patch requires sign off still
Created attachment 6904 [details] [review] BUG 6098 zebra conf NSB NSE indexed as space Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Created attachment 6905 [details] [review] Bug 6098 Followup -- doesn't replace NSB/NSE by space On any page, NSB/NSE characters are replaced by space in Output.pm. It isn't a good thing: - It isn't necessary since Zebra indexer handle properly those characters as delimiter. - It isn't necessary for display since XSL file already do that. - It isn't efficient since a regex is done on every whole page sent by Koha, not only on biblio record part. - It removes an information that librarian want to keep in their biblio records: any record edited in Koha loose its NSB/NSE characters. Signed-off-by: Frédéric Demians <f.demians@tamil.fr> Signed-off-by: Chris Cormack <chrisc@catalyst.net.nz>
Simple changes to handling of NSB and NSE characters; removes regex processing and adds to Zebra config. Marking as Passed QA
patch pushed, please test QA COMMENT: seing comments 2 and 3, shouldn't the line map (\x88.*\x89)) @ also be removed from /etc/zebradb/etc/word-phrase-utf.chr file ?