Bug 7518

Summary: searches with quotation marks don't work
Product: Koha Reporter: Melia Meggs <melia>
Component: SearchingAssignee: Fridolin Somers <fridolin.somers>
Status: CLOSED FIXED QA Contact: Marcel de Rooy <m.de.rooy>
Severity: normal    
Priority: P3 CC: bob, dcook, dubyk, gmcharlt, jcamins, katrin.fischer, koha.aixmarseille, kyle.m.hall, kyle, m.de.rooy, marjorie.barry-vila, mathsabypro, paul.poulain, tomascohen
Version: 3.14   
Hardware: All   
OS: All   
See Also: http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=12443
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=18294
Change sponsored?: --- Patch complexity: Trivial patch
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Bug Depends on:    
Bug Blocks: 9234    
Attachments: Proposed patch
Proposed patch (revisited)
[PATCH] Bug 7518: searches with quotation marks don't work
Proposed patch
Bug 7518: searches with quotation marks don't work
screenshot of search configuration used for testing
Bug 7518: searches with quotation marks don't work

Description Melia Meggs 2012-02-09 21:19:25 UTC
A search term without quotation marks returns the expected results.

The same search term with quotation marks returns no results.

I expected Koha to ignore the quotation marks and return results anyway...
Comment 1 Jared Camins-Esakov 2012-02-09 23:12:51 UTC
I can confirm this is a bug with ICU enabled (this may be configuration-specific). It does not seem to affect non-ICU installations, however.
Comment 2 Serhij Dubyk 2012-02-10 10:03:34 UTC
(In reply to comment #1)
> I can confirm this is a bug with ICU enabled (this may be
> configuration-specific). It does not seem to affect non-ICU installations,
> however.

Sorry for oftop, but I really wonder how to use ICU in Koha and that this can be a benefit (or not), example for libraries in Cyrillic?
Found http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id 3216
but as it applied to have already installed system?
Comment 3 Melia Meggs 2012-02-23 19:50:29 UTC
I have some more info.  Brendan did some more testing and found that this bug seems to affect libraries both with and without ICU enabled.  BUT it only happens when you have the query autotruncate sys pref set to "only if the * is added."
Comment 4 Mathieu Saby 2012-12-06 23:46:47 UTC
Hello
I confirm the bug with ICU and QueryStemming activated.
Very annoying...


M. Saby
Rennes 2 University
Comment 5 Fridolin Somers 2012-12-14 09:50:57 UTC Comment hidden (obsolete)
Comment 6 Kyle M Hall 2012-12-14 14:45:32 UTC
The comment on the new line needs to be fixed.

Also, if you can provide a detailed plan to reproduce the bug, that would be great. I've so far failed to reproduce it. I have ICU enabled and all the other sysprefs set up, but no error.
Comment 7 Fridolin Somers 2012-12-14 16:45:06 UTC Comment hidden (obsolete)
Comment 8 Mathieu Saby 2012-12-18 09:48:32 UTC
Thank you Fridolyn
Your patch improves Koha's behavior, but I am not sure I can sign off.
Double quotes don't break searchs anymore, but they don't force Koha to consider the query as an "exact expression", as I think they should do.

Ex :
One book has this title : La mémoire ou l'oubli
and this autor : Alain Bosquet

After applying yout patch, if I search "La mémoire ou l'oubli", with double quotes, I find the book.
But if I search "L'oubli ou la mémoire" I find it too...
And I can also find the book with "mémoire Bosquet" within double quotes.


M. Saby
Rennes 2 University
Comment 9 Fridolin Somers 2012-12-18 10:58:35 UTC
(In reply to comment #8)
> Thank you Fridolyn
> Your patch improves Koha's behavior, but I am not sure I can sign off.
> Double quotes don't break searchs anymore, but they don't force Koha to
> consider the query as an "exact expression", as I think they should do.
Ok, but this is not the same problem.
It can be asked as enhancement in a new bug.
Comment 10 Mathieu Saby 2012-12-18 12:01:10 UTC
Maybe it could be seen as an other problem, but I am not sure whether it be an enhancement or a correction : with some settings, double quotes are forcing Koha to treat search terms as an exact phrase, but with other settings it is not the case.
And I could not information regarding the use of quotes in the manual, so I don't know what Koha is "supposed" to do...

M. Saby
Comment 11 Fridolin Somers 2012-12-18 12:50:52 UTC
(In reply to comment #10)
CCL syntaxe for an exact match is 'exact' after the index ; it means phrase structure and complete field.
The use of double quotes for exact match is a Google search engine behavior.
I can be very usefull, but maybe hard to manage in actual C4::Search.
Comment 12 Jared Camins-Esakov 2012-12-18 12:54:41 UTC
(In reply to comment #11)
> (In reply to comment #10)
> CCL syntaxe for an exact match is 'exact' after the index ; it means phrase
> structure and complete field.
> The use of double quotes for exact match is a Google search engine behavior.
> I can be very usefull, but maybe hard to manage in actual C4::Search.

Very hard (read: impossible) if we want to keep using CCL2RPN. Fortunately, there's a better way! Bug 9239 introduces a new query parser that uses quotes for phrases.
Comment 13 Mathieu Saby 2012-12-18 13:01:03 UTC
OK. So I sign this bug off (and correct a typo in comments : brakes -> breaks)
Comment 14 Mathieu Saby 2012-12-18 13:02:14 UTC Comment hidden (obsolete)
Comment 15 Marcel de Rooy 2013-02-08 09:27:02 UTC
QA comment:
I would welcome an improved handling of quotes. But look at this simple test:

Searching for "vermeer" did work without this patch but now gives:
Can't call method "sort" on an undefined value  t /usr/share/koha/testclone/C4/Search.pm line 407.
From the zebra log:
WARNING: query problem with kw,wrdl= \\"vermeer\\"  ZOOM error 10014 "CCL parsing error" from diag-set 'ZOOM' at /usr/share/koha/testclone/C4/Search.pm line 359

Any chance of different results due to different Zebra config under MARC21/UNIMARC? 

Failed QA
Comment 16 Fridolin Somers 2013-03-07 15:15:00 UTC Comment hidden (obsolete)
Comment 17 Koha Team AMU 2013-03-18 13:25:20 UTC
Test plan :
- Set QueryWeightFields off
- Perform a serch on two words where you have results, like : centre "ville
=> you get results
- Set QueryWeightFields on
- Perform same serch
=> you get the same results

Patch applies
Comment 18 Paul Poulain 2013-03-18 13:26:49 UTC
Patch tested with a sandbox, by Leila <koha.aixmarseille@gmail.com>
Comment 19 Paul Poulain 2013-03-18 13:27:13 UTC Comment hidden (obsolete)
Comment 20 Katrin Fischer 2013-03-31 19:50:17 UTC
The change looks safe, but I can't see any problems with quotation marks in master. Maybe I didn't get the configuration right?
Comment 21 Kyle M Hall 2013-04-01 12:36:59 UTC
Do you have ICU enabled?

(In reply to comment #20)
> The change looks safe, but I can't see any problems with quotation marks in
> master. Maybe I didn't get the configuration right?
Comment 22 Katrin Fischer 2013-04-02 19:47:30 UTC
I now tested this with and without ICU and both times, I couldn't find any problems. The change doesn't look big, but I am a bit puzzled about not being able to reproduce the problem. Maybe another patch already fixed it?
Please retest on current master.
Comment 23 Katrin Fischer 2013-04-02 19:48:15 UTC
Created attachment 17150 [details]
screenshot of search configuration used for testing
Comment 24 Mathieu Saby 2013-04-02 21:01:31 UTC
Hello
The problem described in this bug seems to be fixed in master. (But it is still present in our preproduction Koha, rebased on 10th of March)

UNIMARC db, ICU
QueryWeightFields Enable
Queryparser Don'try to use

Search "prix du blé" => 1 result (that's correct)
Request : 22:55:02-02/04 zebrasrv(4) [request] Search biblios OK 1 1 1+0 RPN @attrset Bib-1 @attr 1=1016 @attr 4=6 @attr 5=1 "prix du blé"


M. Saby
Rennes 2 university
Comment 25 Fridolin Somers 2013-04-04 13:14:23 UTC
(In reply to comment #24)
> Search "prix du blé" => 1 result (that's correct)
Try with a quote into the search terms : prix du "blé
Comment 26 Fridolin Somers 2013-04-04 13:15:52 UTC
(In reply to comment #23)
> Created attachment 17150 [details]
> screenshot of search configuration used for testing

Try without QueryAutoTruncate.
Comment 27 Katrin Fischer 2013-04-04 14:10:35 UTC
I can't try without - witout search doesn't work at all with ICU :)
See bug 9976
Comment 28 Mathieu Saby 2013-04-04 15:15:10 UTC
I tested "prix du blé", "prix du "blé", prix du "blé, "blé du prix", "blé du "prix", blé du "prix"

=> everything is working in master, without your patch (same number of results)

Mathieu
Comment 29 Katrin Fischer 2013-04-04 15:18:17 UTC
I could imagine this is a side effect of the work on the query parser. Maybe we should test if a patch is needed for 3.10.x still?
Comment 30 Fridolin Somers 2013-04-04 16:03:42 UTC
(In reply to comment #29)
> I could imagine this is a side effect of the work on the query parser. Maybe
> we should test if a patch is needed for 3.10.x still?
Yes I think so.
Comment 31 Katrin Fischer 2013-04-07 07:37:11 UTC
Resetting version.
Comment 32 Fridolin Somers 2013-07-10 14:22:58 UTC
This is still a bug in current master :
- with or without ICU
- with or without UseQueryParser syspref
- only with QueryWeightFields enabled
Comment 33 Mathieu Saby 2013-07-10 14:38:20 UTC
That's puzzling... Some months ago I was sure it was fixed.
So you think a patch is still needed ?

Mathieu
Comment 34 Fridolin Somers 2013-07-11 11:13:06 UTC
> So you think a patch is still needed ?
Yes.

Looking at screenshot of configuration, I think the bug appears only with QueryStemming off. In fact, to test : disable all query sysprefs unless QueryWeightFields and set QueryAutoTruncate to "only if * is added".

I set back to signed off.
A second signoff is welcomed.
Comment 35 Kyle M Hall 2013-07-12 14:16:11 UTC
Created attachment 19596 [details] [review]
Bug 7518: searches with quotation marks don't work

A search term without quotation marks returns the expected results.
The same search term with quotation marks returns no results.
Koha should ignore the quotation marks and return results anyway.

This appears when  QueryWeightFields syspref is activated. This feature composes a complex CCL query using double quotes around search words.
This patch simply replaces double quotes in search words by a space.

Test plan :
- Set QueryWeightFields off
- Perform a serch on two words where you have results, like : centre "ville
=> you get results
- Set QueryWeightFields on
- Perform same serch
=> you get the same results

Signed-off-by: Leila <koha.aixmarseille@gmail.com>
Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 36 Galen Charlton 2013-07-24 17:36:08 UTC
I'm having difficulty reproducing this bug.  For those who can, perchance are there any ZOOM error messages associated with it showing up in the Apache error log?
Comment 37 Fridolin Somers 2013-07-31 17:19:55 UTC
(In reply to Galen Charlton from comment #36)
> I'm having difficulty reproducing this bug.  For those who can, perchance
> are there any ZOOM error messages associated with it showing up in the
> Apache error log?

The best way is to look at Zebra server output.
- launch Zebra server manually : zebrasrv -f /home/koha/etc/koha-conf.xml
- Set syspref QueryWeightFields on
- Perform a search on two words. For example : centre ville :
- Look at zebrasrv log :
19:15:18-31/07 zebrasrv(4) [request] Search biblios OK 0 1 1+0 RPN @attrset Bib-1 @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 "centre ville" @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 "centre ville" @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 "centre ville" @attr 4=6 @attr 9=14 @attr 2=102 "centre ville"
- Perform a search with a double quote. For example : centre "ville :
- Look at zebrasrv log :
19:15:13-31/07 zebrasrv(3) [request] Search biblios OK 0 1 1+0 RPN @attrset Bib-1 @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 "centre ville or ti,ext,r2=centre ville" @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 "centre ville or wrdl,r9=centre ville"

=> You see that with double quote, there is some unconverted CCL into PQF query : or wrdl,r9=
Comment 38 Fridolin Somers 2013-11-12 09:50:12 UTC
Can it be pushed in master ?
Comment 39 Galen Charlton 2013-12-17 16:40:34 UTC
Pushed to master, along with a regression test.  Thanks, Fridolin!
Comment 40 Fridolin Somers 2014-01-27 12:35:50 UTC
Patch pushed to 3.14.x, will be in 3.14.3.
Comment 41 Tomás Cohen Arazi 2014-02-07 17:13:33 UTC
This patch has been pushed to 3.12.x, will be in 3.12.11.

Thanks Fridolin!
Comment 42 David Cook 2014-06-30 08:23:37 UTC
(In reply to Jared Camins-Esakov from comment #12)
> (In reply to comment #11)
> > (In reply to comment #10)
> > CCL syntaxe for an exact match is 'exact' after the index ; it means phrase
> > structure and complete field.
> > The use of double quotes for exact match is a Google search engine behavior.
> > I can be very usefull, but maybe hard to manage in actual C4::Search.
> 
> Very hard (read: impossible) if we want to keep using CCL2RPN. Fortunately,
> there's a better way! Bug 9239 introduces a new query parser that uses
> quotes for phrases.

As Jared has mentioned, I think the use of double quotes to signify a phrase/exact search would also be difficult to do while using CCL2RPN.

However, as some people have mentioned, it seems that Koha has had this ability in the past...

This is, in part, because the default Koha search is a kw,wrdl search (i.e. a Word list search on the Keyword index). 

"The structure attribute values Word list (6) is supported, and maps to the boolean AND combination of words supplied. The word list is useful when Google-like bag-of-word queries need to be translated from a GUI query language to PQF."
Source: http://www.indexdata.com/zebra/doc/querymodel-rpn.html

In other words, doing a regular search for "professional responsibility" would not do a phrase search. Rather, it would do a search like this: "professional" AND "responsibility". In some cases, this will return the exact same results as a phrase search would, but technically speaking it's not the same because proximity and order of the words to each other doesn't take place in a "Word list" type search.

The other reason why it seemed that we had this functionality in the past is probably due to conflicting system preferences which accidentally turned off QueryWeightFields. 

Truncation and QueryWeightFields really do not play well together. I am endeavouring to fix that. I'll open other bugs to fix that though...
Comment 43 David Cook 2014-06-30 08:26:09 UTC
It's also worth mentioning that this patch doesn't work very well when doing a regular (i.e. 'Library Catalog' or 'kw') search, when using "QueryWeightFields", and when NOT using QueryAutoTruncate.

It appears to be an issue with the quoting of strings in _build_weighted_query.

I won't bother sending a follow-up here, as I'm planning to refactor _build_weighted_query.

NOTE: If you are using QueryAutoTruncate (ie all queries are autotruncated) then QueryWeightFields will be ignored (even though it's on by default), so you'll probably have no problems (although you also won't have any relevance searching).
Comment 44 Fridolin Somers 2014-07-02 07:26:15 UTC
All this needs to be considered in the QueryParser.