Bug 5534 - Koha mapping punctuation to space for indexing/searching can be confusing
Summary: Koha mapping punctuation to space for indexing/searching can be confusing
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Zebra (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal with 1 vote (vote)
Assignee: Galen Charlton
QA Contact: Bugs List
URL:
Keywords:
Depends on:
Blocks: 9234
  Show dependency treegraph
 
Reported: 2010-12-22 00:01 UTC by Nicole C. Engard
Modified: 2023-12-01 01:37 UTC (History)
17 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Nicole C. Engard 2010-12-22 00:01:39 UTC
When you're doing a search for a title like "Harrison's Internal Medicine" you can find it if you search for "harrison" or "harrison's" but not "harrisons" - the punctuation is required.  This seems a bit odd since most search engines don't require that you enter the punctuation.
Comment 1 Ian Walls 2011-12-02 21:14:17 UTC
Unfortunately, the Zebra charmap files do not have a command for mapping a character to 'null'.  Right now, apostrophe is mapped to 'space', which serves to break up words.  So searching for "harrison's internal" searches for three words: "harrison", "s" and "internal".

We may be able to explicitly map "'s" to "s"... but then we'd need to do that for EVERY possible case where apostrophe is used....

ICU might fix this yeah?
Comment 2 Kyle M Hall 2012-02-08 15:57:40 UTC
Has anyone tried contacting Index Data about the inability to map a character to null?
Comment 3 wajasu 2012-09-12 01:14:11 UTC
Not sure if this applies.  I was doing search testing even with fuzzy search.

I tried "1000 things i love about" with no results, though the title was "1,000 things i love".  I t only found "1,000 things i love". 
So comma's or other locale specific formatting in words that are all digits might be a similiar use case.
Comment 4 Rafa Saad 2012-11-28 17:00:23 UTC
When we make a search for titles like "Você viu meu pai por aí?", that has a "?", we have results only with a search for "Você viu meu pai por aí" (without "?"). The same problem when the search term has "!". For dots or commas this problem doesn't occurs.
Comment 5 Jared Camins-Esakov 2012-11-28 17:02:01 UTC
(In reply to comment #4)
> When we make a search for titles like "Você viu meu pai por aí?", that has a
> "?", we have results only with a search for "Você viu meu pai por aí"
> (without "?"). The same problem when the search term has "!". For dots or
> commas this problem doesn't occurs.

You will be glad to know that the search rewrite (which is already underway) will fix this particular issue.
Comment 6 Nicole C. Engard 2012-12-10 15:03:26 UTC
I'm wondering if this also includes the ampersand (&) ? I'm having an issues with searching for a title that includes it - but wasn't sure if it was a new bug.
Comment 7 Isaac Brodsky 2013-03-07 16:12:47 UTC
Just to add my two cents, a user could want to search for punctuation. (So searching for "C++" actually searches for "C", which is undesirable.) Is this intended?
Comment 8 kholt 2014-01-09 22:20:06 UTC
Having this problem with Xmen and X-men in our catalog  (3 results versus 235).  Actually X men and X-men gives same number of results so maybe that's the best we can do for now?
Comment 9 Christopher Brannon 2015-05-14 17:51:20 UTC
We're seeing this when we do searches between the word "don't" and "dont".  This is a real, ongoing problem.  I'm wondering if there is anything that can be done outside of waiting for ElasticSearch?
Comment 10 Tomás Cohen Arazi 2015-05-14 18:07:36 UTC
We should fix this taking advantage of DOM: i.e. generating indexes for the strings with punctuation removed. It could be tricky in some scenarios, but it should work.
Comment 11 Katrin Fischer 2015-05-15 22:26:17 UTC
I think also ICU could help/work here - I remember we tested both, mapping - to a space and to nothing, only thing that didn't seem possible is doing both at the same time.
Comment 12 dguidry 2017-08-18 12:29:54 UTC
Is there any update in regards to this particular issue? I'm running Koha 17.05 and are experiencing the same problem.

One example is we have an entry for "Expositor's Dictionary". The only way it will return a result is by searching for "expositor" or "expositor's", but the average person seems to type "expositors" and it returns no results. A typical search engine in 2017 is generally able to handle this, so this is the prevailing user expectation these days.

Are there any plans to resolve this sort of usability issue? We've had multiple people report this problem over the last year when searching for various items in our catalog that contain an apostrophe. If there's no fix in the works, is there a decent workaround that can be implemented in the meantime to smooth this over for our users?
Comment 13 Katrin Fischer 2017-08-18 13:05:45 UTC
Hi, I suggest filing a separate bug with your example - Zebra is able to do things like that, but there are known issues with quotes and similar. Sometimes it also depends on configuration (ICU vs. grs-1, installation choices, etc...).
Please include information about how you installed Koha and if you changed defaults etc. on installation if possible.
Comment 14 Katrin Fischer 2017-08-18 13:09:24 UTC
Sorry, I see now that this here all started with the discussion about apostrophes - just a lot of information and other problems have been added since. Maybe splitting this into several bugs would still be good - we can link them all to this one.
Comment 15 J Schmidt 2017-08-18 15:31:20 UTC
Is this bug specifically tied to Zebra indexing or with this also be the case with ElasticSearch as Koha users move towards that? Thanks.
Comment 16 Katrin Fischer 2017-08-18 15:42:55 UTC
I think I'd put Elasticsearch related issues on a separate bug - most of the discussion here applies to Zebra and both have different configuration files.
Comment 17 dguidry 2017-08-21 14:21:08 UTC
We originally installed Koha around May or June of 2016 and would have been on the 3.22 version that was released at that time. We used defaults for everything as far as I remember and it appears that we're using Zebra. Is it possible to switch to ElasticSearch and is it fairly involved to do so? Are there any other considerations to take into account before doing so? Is ElasticSearch production-ready?

Thanks!
Comment 18 Nick Clemens (kidclamp) 2017-08-22 11:16:08 UTC
(In reply to dguidry from comment #17)
> We originally installed Koha around May or June of 2016 and would have been
> on the 3.22 version that was released at that time. We used defaults for
> everything as far as I remember and it appears that we're using Zebra. Is it
> possible to switch to ElasticSearch and is it fairly involved to do so? Are
> there any other considerations to take into account before doing so? Is
> ElasticSearch production-ready?
> 
> Thanks!

Elasticsearch is still in testing phase I would say,there are a few things that need to be cleaned up to be production ready, most of them are small issues, but together they would cause some pain points. Most of them relate to indexing configuration and Koha expectations - if you don't define expected fields sortable - sorting causes a failed search for example

Setting it up should be fairly straight forward - Mirko has put together a 'koha-elasticsearch' package which will install needed dependencies for koha, and you will need to install Java and Elasticsearch itself.

I will work on some of the issues in the coming weeks and update the status page on the wiki and send something to the list.

We do need testers and are working on enabling ES on some sandboxes and will update when that work is ready.
Comment 19 David Cook 2023-12-01 01:37:58 UTC
For folk using Zebra in English, QueryStemming can help with this, although QueryStemming only works if automatic QueryTruncation isn't turned on.