Bug 9729 - Unable to use IT search terms such as C#, .NET, C++ in searching
Summary: Unable to use IT search terms such as C#, .NET, C++ in searching
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low normal (vote)
Assignee: Galen Charlton
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2013-02-28 14:47 UTC by David Hartman
Modified: 2023-12-01 01:45 UTC (History)
5 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description David Hartman 2013-02-28 14:47:55 UTC
Type in the phrase "C#" in a search field, and search results has every record with a c.

Type in the phrase "C++" in a search field, and search results has every record with a c.


Type in ".NET", and you get more results than .NET books.
Comment 1 Katrin Fischer 2014-09-24 10:05:55 UTC
This is still an issue. 
Tested in 3.16.3 with GRS-1 indexing.
Comment 2 Marjorie Barry-Vila 2017-09-14 15:43:15 UTC
Still an issue in 16.11.

Marjorie
Comment 3 Katrin Fischer 2019-04-27 21:26:14 UTC
Would be interesting to test with Elasticsearch.
Comment 4 Marjorie Barry-Vila 2021-06-16 20:11:17 UTC
Still valid with Zebra and ElasticSearch in 20.05

Marjorie
Comment 5 David Cook 2021-06-17 00:10:28 UTC
(In reply to Marjorie Barry-Vila from comment #4)
> Still valid with Zebra and ElasticSearch in 20.05
> 
> Marjorie

It would be interesting to know if it's CHR or ICU indexing for Zebra.

These symbols get stripped out in the normalization process. With phrases in ICU, I see we're arbitrarily removing all punctuation. With words in ICU, it looks like we might only be removing punctuation that is preceded by whitespace (which seems to imitate the example chain.xml from https://software.indexdata.com/yaz/doc/yaz.pdf). So that might be OK. In fact I should be able to test that...

With CHR indexing, symbols get converted into spaces, so it won't work.
Comment 6 David Cook 2021-06-17 00:12:57 UTC
Hmm nope...

echo "C++" | yaz-icu -c words-icu.xml
1 1 'c+' 'C+'
2 1 '+' '+'

echo "C#" | yaz-icu -c words-icu.xml
1 1 'c' 'C'

echo ".NET" | yaz-icu -c words-icu.xml
1 1 'net' 'NET'
Comment 7 David Cook 2021-06-17 00:25:51 UTC
After some experimenting, it seems YAZ ICU will tokenize based on the "+" without any normalization when using the "line" tokenize rule:

echo -n "C++" | yaz-icu -c chain.xml
1 1 'c+' 'C+'
2 1 '+' '+'

echo -n "C#" | yaz-icu -c chain.xml
1 1 'c#' 'C#'

echo -n ".NET" | yaz-icu -c chain.xml
1 1 '.net' '.NET'

I wonder if that's a bug in YAZ because it doesn't do that for all punctuation/symbols... 

echo -n 'C--' | yaz-icu -c chain.xml
1 1 'c--' 'C--'

echo -n 'C???' | yaz-icu -c chain.xml
1 1 'c???' 'C???'
Comment 8 David Cook 2021-06-17 00:27:54 UTC
(In reply to David Cook from comment #5)
> With words in ICU, it
> looks like we might only be removing punctuation that is preceded by
> whitespace (which seems to imitate the example chain.xml from
> https://software.indexdata.com/yaz/doc/yaz.pdf). 

Ooops. I misread that. Actually, it just strips both whitespace and punctuation regardless of position.

Yeah, I don't think this will work with Zebra, since both the word and phrase index registers are supposed to be normalized.

I am curious how Google does it. Historically, Google used to normalize its queries and strip out symbols, but the past few years it seems to be more intelligent (it will even convert "+" into "plus" so there is some good AI at work there I reckon).
Comment 9 David Cook 2021-06-17 01:13:12 UTC
(In reply to Katrin Fischer from comment #3)
> Would be interesting to test with Elasticsearch.

I agree.

It looks like a bit of a known issue: https://stackoverflow.com/questions/63477160/elastic-analyzer-to-enable-matching-searches-such-as-c-c-a
https://discuss.elastic.co/t/searching-and-indexing-of-c-c-net-asp-net-doesnt-work/4070/3
Comment 10 David Cook 2021-06-17 01:22:36 UTC
Oh I've had some fun playing with ICU...

chain.xml:
<icu_chain locale="">
  <tokenize rule="l"/>
  <transliterate rule="[:Punctuation:] } [:WhiteSpace:] > ''"/>
  <transform rule="[:WhiteSpace:] Remove "/>
  <display/>
  <casemap rule="l"/>
</icu_chain>

echo -n '.NET. test' | yaz-icu -c chain.xml
1 1 '.net'' '.NET''
2 1 'test' 'test'

--
Here we tokenize based on the line break (ie space), and then we perform our transliteerate and transform rules as per http://userguide.icu-project.org/transforms/general. 

With the transliterate, we can use the following syntax:

"before_context { text_to_replace } after_context > completed_result | result_to_revisit ;"

So here the "text_to_replace" is the [:Punctuation:] and the "after_context" is [:WhiteSpace:], and the completed result is transliterating the punctuation into nothing. 

So we trim the "." from the end of NET but we don't trim the "." from the start. 

Of course, that doesn't really work in practice, because it misses sooo many other scenarios:

echo -n 'Was that a good idea?' | yaz-icu -c chain.xml
1 1 'was' 'Was'
2 1 'that' 'that'
3 1 'a' 'a'
4 1 'good' 'good'
5 1 'idea?' 'idea?'

I'm not really sure how to solve this problem in an efficient way. We could just map "C#", "C++", and ".NET" to "csharp", "cplusplus', and 'dotnet', but that's not a very scalable or comprehensive solution for all Koha users.
Comment 11 David Cook 2021-06-17 01:26:18 UTC
Of course, some characters also have special significance in the CCL query language as well, so even getting them to Zebra would be part of the first step.

Not a simple problem by any stretch.
Comment 12 David Cook 2023-12-01 01:45:35 UTC
It would be interesting to look at this in an Elasticsearch context...