Bug 28884

I get a result if I put a space between my last word and the “?” symbol, even tho there is no space in the title of the document.

How come we have this issue with “?” but not with “!” ?

They've already fixed the exclamation point on bug 28316 :)

I found this explanation about the question mark:

The question mark is a special character in es query strings. For example,
if you search "me?t", this means that the results we seek must contain a
word that contain "At Most" one character between "me" and "t". So the
words "meet", "meat" or "met" will match the query.

So maybe a fix is not so easy, as we either make it ignore ? or keep the feature?

Just verified:

Pe?l finds Perl books and Per? does as well. Perl? won't.

Observations:

1. It happens that question mark (‘?’) is contained in bibliographic data (e.g. in title).

2. It happens that user perform searches with that contain question mark (e.g. searching with a title taken verbatim).

3. In ES queries, in particular in query_string query, question mark has a special meaning – substitutes one character (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-wildcard).  Question mark has also special meaning in regexp inside quiery_string (https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_regular_expressions).

4. Ordinary users are not aware of this special function of the question mark and, as a result, receive no results (e.g. while searching for title like "Will libraries survive? : the future of public libraries").

So, the question mark should be removed from query term (clean_search_term) or -- better -- escaped in the same way as slash is escaped (_query_regex_escape_process). A patch will follow.

Created attachment 147161 [details] [review]
[PATCH] Bug 28884: ElasticSearch: Question mark in title search returns no results

1. It happens that question mark (‘?’) is contained in bibliographic
data (e.g. in title).

2. It happens that user perform searches with that contain question
mark (e.g. searching with a title taken verbatim).

3. In ES queries, in particular in query_string query, question
mark has a special meaning – substitutes one character
(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#query-string-wildcard).
Question mark has also special meaning in regexp inside
quiery_string
(https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html#_regular_expressions).

4. Ordinary users are not aware of this special function of the
question mark and, as a result, receive no results (e.g.
while searching for title like "Will libraries survive? :
the future of public libraries").

Test plan:
==========
1. Have biblio record with title like
   245 10 $a Will libraries survive? : $b the future of public libraries.
   (= with question mark attached to a word) in Koha instance with ES.
2. Search for: Will libraries survive? the future of public libraries
   You should get no results.
3. Apply the patch, controll that QueryRegexEscapeOptions is set to
   Escape.
4. Repeat the search.  You should get the record.

This was "solved" with a hack on bug 31213

If I understand what this patch is doing, you are escaping the question mark so it no longer operates as a special character?

This is not a bad idea, however, it needs to be documented with a comment in the code, as well as in the system preference. I think it would actually be better as its own system preference.

(In reply to Nick Clemens from comment #7)
> This was "solved" with a hack on bug 31213
> 
> If I understand what this patch is doing, you are escaping the question mark
> so it no longer operates as a special character?
> 
> This is not a bad idea, however, it needs to be documented with a comment in
> the code, as well as in the system preference. I think it would actually be
> better as its own system preference.

Nick, I'm aware of your 'hack'--now, I'm unable to verify if quickly, but as far as I remember it didn't solve this case.  If it did, there would be no point for me to dig into this.

If I remember well, the patch intends to treat the question mark, in the function _query_regex_escape_process, in exactly the same way as it is done with slash: instead of escaping/unescaping only '/': (?=/) it escapes also '?': (?=[/\?]).  Maybe I could extend the existing comment, but I don't feel like this is a candidate for a new syspref...?

(In reply to Janusz Kaczmarek from comment #8)
> (In reply to Nick Clemens from comment #7)
> > This was "solved" with a hack on bug 31213
> > 
> > If I understand what this patch is doing, you are escaping the question mark
> > so it no longer operates as a special character?
> > 
> > This is not a bad idea, however, it needs to be documented with a comment in
> > the code, as well as in the system preference. I think it would actually be
> > better as its own system preference.
> 
> Nick, I'm aware of your 'hack'--now, I'm unable to verify if quickly, but as
> far as I remember it didn't solve this case.  If it did, there would be no
> point for me to dig into this.
> 
> If I remember well, the patch intends to treat the question mark, in the
> function _query_regex_escape_process, in exactly the same way as it is done
> with slash: instead of escaping/unescaping only '/': (?=/) it escapes also
> '?': (?=[/\?]).  Maybe I could extend the existing comment, but I don't feel
> like this is a candidate for a new syspref...?

I think it would be nice if we could keep the truncation feature without requiring to escape it. It's easy to explain to people to use ?, but much harder to explain \? - also quite a change in behavior at this point maybe?

I was wondering, if ? replaces one or more characters... why does it not work as a wild card if there really is a question mark?

(In reply to Katrin Fischer from comment #9)
> (In reply to Janusz Kaczmarek from comment #8)
> > (In reply to Nick Clemens from comment #7)
> > > This was "solved" with a hack on bug 31213
> > > 
> > > If I understand what this patch is doing, you are escaping the question mark
> > > so it no longer operates as a special character?
> > > 
> > > This is not a bad idea, however, it needs to be documented with a comment in
> > > the code, as well as in the system preference. I think it would actually be
> > > better as its own system preference.
> > 
> > Nick, I'm aware of your 'hack'--now, I'm unable to verify if quickly, but as
> > far as I remember it didn't solve this case.  If it did, there would be no
> > point for me to dig into this.
> > 
> > If I remember well, the patch intends to treat the question mark, in the
> > function _query_regex_escape_process, in exactly the same way as it is done
> > with slash: instead of escaping/unescaping only '/': (?=/) it escapes also
> > '?': (?=[/\?]).  Maybe I could extend the existing comment, but I don't feel
> > like this is a candidate for a new syspref...?
> 
> I think it would be nice if we could keep the truncation feature without
> requiring to escape it. It's easy to explain to people to use ?, but much
> harder to explain \? - also quite a change in behavior at this point maybe?
> 
> I was wondering, if ? replaces one or more characters... why does it not
> work as a wild card if there really is a question mark?

Well, IMHO it will be not so easy to explain to an ordinary user (who usually will not read complicated introductory texts, and what we are discussing is a very special case) how '?' functions with ES.  Moreover, I expect that an ordinary user will rather use '?' as a verbatim '?' (from copy-paste) than as a wildcard.  So, I expect (and have already experienced) alarms from the users who are not able to find a book (with a title containing '?', copied from other data source) in the local catalog even if it is there. 

If I understand well, in query_string ES query, '?' stands for a single character. But, at the same time, after a normal text analysis of a string containing a question mark attached at the end of the word, the '?' character is removed from what is stored in the index.  So, the query 'are?' will not find an original string 'are?' but it would find for instance 'ares', 'arer', 'area' etc.

(Similarly, '/' has a special meaning in ES but at the same time can be put unconsciously by an ordinary user as a part of query string -- and hence, if I get it right, the option QueryRegexEscapeOptions of escaping it.)

? and * are quite common for truncation and databases and the like, people have been writing tutorials on Elasticsearch etc. like:
https://drive.google.com/file/d/17-YVjG6IJKeDmDCKuF7ZYvcTfzDYyxwa/view 
That's also been promoted on the BywaterSolutions blog. 

So next question would be, why * and ? are handled differently... could the behaviour of Elasticsearch removing the ending ? be changed?

(In reply to Katrin Fischer from comment #11)
> ? and * are quite common for truncation and databases and the like, people
> have been writing tutorials on Elasticsearch etc. like:
> https://drive.google.com/file/d/17-YVjG6IJKeDmDCKuF7ZYvcTfzDYyxwa/view 
> That's also been promoted on the BywaterSolutions blog. 
> 
> So next question would be, why * and ? are handled differently... could the
> behaviour of Elasticsearch removing the ending ? be changed?

The fundamental differences between ? and * are:

1. * very rarely would be present in ordinary users' copy-paste queries,
2. even if, * means '0 or more characters' (like /.*/ in Perl) -- so it would not create problems -- whereas ? means _exactly_ one character (like Perlish /./) -- and therefor is quite problematic in the context we are in.

Still, I would not expect from the users of the catalogue the knowledge about quite advanced, wildcarded ES queries... s/he is usually is not interested in what is in the back-end.  Hence, I would prefer to prepare Koha to first "understand" a normal user instead of advanced search engines operator.

And still, the basic scenario is: the user does copy-paste search and does not get the result -- which for him/her (and therefor for me) is not acceptable.

(In reply to Janusz Kaczmarek from comment #8)
> (In reply to Nick Clemens from comment #7)
> > This was "solved" with a hack on bug 31213
> 
> Nick, I'm aware of your 'hack'--now, I'm unable to verify if quickly, but as
> far as I remember it didn't solve this case.  If it did, there would be no
> point for me to dig into this.

Ah, so it doesn't solve this issue because we don't index the a and b together by default. So when you search for: "Will libraries survive? the future of public libraries"
The quoted version fails, because no single field contains that string.

If you change or add a title index with 245(ab) then the "hack" does work. In fact, it's probably a good idea because any quoted 'full' title search including a subtitle will fail otherwise

(In reply to Nick Clemens from comment #13)
> If you change or add a title index with 245(ab) then the "hack" does work.
> In fact, it's probably a good idea because any quoted 'full' title search
> including a subtitle will fail otherwise

Thank you for this comment.  Yes, reformulating the search field definition in mappings would be of some help (but which search field? 'title' for sure - into 245(ahbnpc) at least - but will it solve all issues?).  

But it would not solve the problem universally.  For instance, having the title in the record: "$a Will libraries survive? : $b the future of public libraries" and querying: "Will libraries survive? future of public libraries" (without 'the' in this case, but in general: missing any element from inside the recorded element) will still give the user no results.  And this is not just for teasing--often the author taken from another data source would be transcribed differently in 245 $c (like omitting 'by' for instance).

I am not against the concept of auto-escaping the question mark, however, I do think it needs to be controlled by a syspref. Possibly with different options for staff and opac. 

The code and the preference description also need to be clear about what they are doing - silently escaping '?' in the code that handles regex searching is confusing

Since this remains a definite usability issue, would it be sufficient to get a fix in place by adding a system preference?

And to clarify what I mean, a new system preference that is just about the ES wildcard/truncation characters and permits different behavior in staff and OPAC?

I'm not sure if I would like to have one preference (even if in two incarnations: OPAC and staff) switching off all the ES wildcard and truncation special characters... From some months of experience with Koha + ES I would say, it is '/' and '?' (sometimes also '!') that are problematic.  There is no problem with most useful '*' or '~' -- even if unconsciously used by a user.  

From my perspective it would be best to have the special meaning of '?', '/', '!' switched off by default, while '*' and '~' enabled.

Tricky question, this. I think I agree with Janusz that it would be more useful for "end users" if we made it possible to search for "Will libraries survive?" than to enable the "me?t" syntax. But opinions can differ on this, as we see, so making it configurable makes sense. And people might want different behaviour in OPAC and staff. 

Just my NOK 0.02.

Yeah this is a tough one. 

Sometimes I think that we shouldn't try to be too smart, and instead just be more honest with end users.

Something like 'Tip: a question mark is a reserved character! To use it, you must escape it like this: title:"Who is there\?"'

When we try to be too smart programmatically, we end up in nightmare situations.

OK, but how about the general idea like this: a "normal" search should take into account only letters and digits and require from the user no special knowledge about the underlying technology, query grammar, expert constructs etc.  In this case a "copy and paste" search will always return some results if there are record matching the search keywords. 

At the same time, in advanced search -- a checkbox to enable ES grammar (i.e. not to strip any punctuation, parentheses, quotes etc.).

Wouldn't it be what an average inexperienced user would expect from a search tool?  What do you think?

(In reply to Janusz Kaczmarek from comment #21)
> OK, but how about the general idea like this: a "normal" search should take
> into account only letters and digits and require from the user no special
> knowledge about the underlying technology, query grammar, expert constructs
> etc.  In this case a "copy and paste" search will always return some results
> if there are record matching the search keywords. 
> 
> At the same time, in advanced search -- a checkbox to enable ES grammar
> (i.e. not to strip any punctuation, parentheses, quotes etc.).
> 
> Wouldn't it be what an average inexperienced user would expect from a search
> tool?  What do you think?

I like the idea of a checkbox or option to enable use of special syntax - if escaping by default I would want some sort of 'i'/info marker next to the search box just so people know

(In reply to Janusz Kaczmarek from comment #21)
> Wouldn't it be what an average inexperienced user would expect from a search
> tool?  What do you think?

The tough part with Koha is that we use the same internal tooling for both the OPAC and the staff interface. We've got 2+ kinds of users with Koha.

What would make sense for the OPAC doesn't necessarily make sense for the staff interface. My librarians get angry at Koha for making their searches imprecise. 

But I agree that for the OPAC something "that just works (like Google)" is generally what people probably want. 

(Of course, I hate using public library OPACs, because they're frequently awful for searching and often give totally irrelevant results despite your search query. But that's a non-Koha rant for another day hehe.)

(In reply to Nick Clemens (kidclamp) from comment #15)
> I am not against the concept of auto-escaping the question mark, however, I
> do think it needs to be controlled by a syspref. Possibly with different
> options for staff and opac. 

Agreed. 

When we're using the Advanced Search, I'd say we should auto-escape all reserved characters, since it's a structured search. 

Since we (rightly or wrongly) allow complex queries in the regular "Simple" Search, we're caught in a bit of a tough place. We're giving users the power to write complex queries... but we can't trust them to escape reserved characters either. It's a bit... absurd really. 

However, for many years, we've had this absurd approach, so for backwards compatibility I think we need to keep it going. 

> The code and the preference description also need to be clear about what
> they are doing - silently escaping '?' in the code that handles regex
> searching is confusing

Agreed. The splinter review looks identical at a glance. If I recall correctly, you can split regex across newlines. Alternatively, we might be able to build it from a series of strings. The current code isn't readable/maintainable.

(In reply to Janusz Kaczmarek from comment #21)
> OK, but how about the general idea like this: a "normal" search should take
> into account only letters and digits and require from the user no special
> knowledge about the underlying technology, query grammar, expert constructs
> etc.  In this case a "copy and paste" search will always return some results
> if there are record matching the search keywords. 

We are pretty heavy-handed in terms of normalization with Zebra, and it gets us into a bit of a situation: bug 9729 

Amusingly, on that bug report I talk about how we should look into the handling of special characters in Elasticsearch.

--

We've never really had a good comprehensive approach to searching. Having 2 search engines has just made it worse. 

I think if we cleaned up the regex escape a bit, then this patch probably is the way to go. We often take the nuclear approach, and the ? mark is probably not the hill I want to die on when it comes to search. 

As I bring more libraries over to Elasticsearch, I'll be scrutinizing Koha's query building/massaging more. I've been thinking a bit how it could be interesting to be more pluggable/modular with this, so that it's easier to fine-tune searching for individual libraries, but that's a future problem...

(In reply to Janusz Kaczmarek from comment #12)
> Still, I would not expect from the users of the catalogue the knowledge
> about quite advanced, wildcarded ES queries... s/he is usually is not
> interested in what is in the back-end.  Hence, I would prefer to prepare
> Koha to first "understand" a normal user instead of advanced search engines
> operator.

One more comment and then I'll be done. (Apologies if my comments are coming across overly negative. I've spent over a decade looking at Koha's search code, and I've had a lot of librarians grumpy with me about Koha's search functionality in that time. More now with the switch to Elasticsearch than ever really...)

The difficulty here is that we're already expecting them to understand OR/AND and () when it comes to Elasticsearch. 

One could argue that we should escape () and wrap every query in double quotes, if we don't think a normal user can understand complex search queries. But that will really irritate power users (on both staff interface and OPAC). 

And that's not what we've done historically. We try to outsmart the user, and it only kind of works. 

Sometimes I think we should just make a full decision one way or another. The Google style search box should probably just be a simple search with auto-escaping for reserved characters. 

Then we could have a Power Search or something like that in the Advanced Search area for power users where we "take off the kid gloves". 

If you use Google as an example, they actually do explain their search query syntax on their Advanced Search page. (Of course, I just found a little bug in Google's Advanced Search. Nothing is perfect...)

I think that's where system preferences can be powerful, and individual libraries could then also analyze the search behaviour of users to see how they should tweak their search settings.