Type in the phrase "C#" in a search field, and search results has every record with a c. Type in the phrase "C++" in a search field, and search results has every record with a c. Type in ".NET", and you get more results than .NET books.
This is still an issue. Tested in 3.16.3 with GRS-1 indexing.
Still an issue in 16.11. Marjorie
Would be interesting to test with Elasticsearch.
Still valid with Zebra and ElasticSearch in 20.05 Marjorie
(In reply to Marjorie Barry-Vila from comment #4) > Still valid with Zebra and ElasticSearch in 20.05 > > Marjorie It would be interesting to know if it's CHR or ICU indexing for Zebra. These symbols get stripped out in the normalization process. With phrases in ICU, I see we're arbitrarily removing all punctuation. With words in ICU, it looks like we might only be removing punctuation that is preceded by whitespace (which seems to imitate the example chain.xml from https://software.indexdata.com/yaz/doc/yaz.pdf). So that might be OK. In fact I should be able to test that... With CHR indexing, symbols get converted into spaces, so it won't work.
Hmm nope... echo "C++" | yaz-icu -c words-icu.xml 1 1 'c+' 'C+' 2 1 '+' '+' echo "C#" | yaz-icu -c words-icu.xml 1 1 'c' 'C' echo ".NET" | yaz-icu -c words-icu.xml 1 1 'net' 'NET'
After some experimenting, it seems YAZ ICU will tokenize based on the "+" without any normalization when using the "line" tokenize rule: echo -n "C++" | yaz-icu -c chain.xml 1 1 'c+' 'C+' 2 1 '+' '+' echo -n "C#" | yaz-icu -c chain.xml 1 1 'c#' 'C#' echo -n ".NET" | yaz-icu -c chain.xml 1 1 '.net' '.NET' I wonder if that's a bug in YAZ because it doesn't do that for all punctuation/symbols... echo -n 'C--' | yaz-icu -c chain.xml 1 1 'c--' 'C--' echo -n 'C???' | yaz-icu -c chain.xml 1 1 'c???' 'C???'
(In reply to David Cook from comment #5) > With words in ICU, it > looks like we might only be removing punctuation that is preceded by > whitespace (which seems to imitate the example chain.xml from > https://software.indexdata.com/yaz/doc/yaz.pdf). Ooops. I misread that. Actually, it just strips both whitespace and punctuation regardless of position. Yeah, I don't think this will work with Zebra, since both the word and phrase index registers are supposed to be normalized. I am curious how Google does it. Historically, Google used to normalize its queries and strip out symbols, but the past few years it seems to be more intelligent (it will even convert "+" into "plus" so there is some good AI at work there I reckon).
(In reply to Katrin Fischer from comment #3) > Would be interesting to test with Elasticsearch. I agree. It looks like a bit of a known issue: https://stackoverflow.com/questions/63477160/elastic-analyzer-to-enable-matching-searches-such-as-c-c-a https://discuss.elastic.co/t/searching-and-indexing-of-c-c-net-asp-net-doesnt-work/4070/3
Oh I've had some fun playing with ICU... chain.xml: <icu_chain locale=""> <tokenize rule="l"/> <transliterate rule="[:Punctuation:] } [:WhiteSpace:] > ''"/> <transform rule="[:WhiteSpace:] Remove "/> <display/> <casemap rule="l"/> </icu_chain> echo -n '.NET. test' | yaz-icu -c chain.xml 1 1 '.net'' '.NET'' 2 1 'test' 'test' -- Here we tokenize based on the line break (ie space), and then we perform our transliteerate and transform rules as per http://userguide.icu-project.org/transforms/general. With the transliterate, we can use the following syntax: "before_context { text_to_replace } after_context > completed_result | result_to_revisit ;" So here the "text_to_replace" is the [:Punctuation:] and the "after_context" is [:WhiteSpace:], and the completed result is transliterating the punctuation into nothing. So we trim the "." from the end of NET but we don't trim the "." from the start. Of course, that doesn't really work in practice, because it misses sooo many other scenarios: echo -n 'Was that a good idea?' | yaz-icu -c chain.xml 1 1 'was' 'Was' 2 1 'that' 'that' 3 1 'a' 'a' 4 1 'good' 'good' 5 1 'idea?' 'idea?' I'm not really sure how to solve this problem in an efficient way. We could just map "C#", "C++", and ".NET" to "csharp", "cplusplus', and 'dotnet', but that's not a very scalable or comprehensive solution for all Koha users.
Of course, some characters also have special significance in the CCL query language as well, so even getting them to Zebra would be part of the first step. Not a simple problem by any stretch.
It would be interesting to look at this in an Elasticsearch context...
I recently noticed this as I was cataloging some books on programming.