A search term without quotation marks returns the expected results. The same search term with quotation marks returns no results. I expected Koha to ignore the quotation marks and return results anyway...
I can confirm this is a bug with ICU enabled (this may be configuration-specific). It does not seem to affect non-ICU installations, however.
(In reply to comment #1) > I can confirm this is a bug with ICU enabled (this may be > configuration-specific). It does not seem to affect non-ICU installations, > however. Sorry for oftop, but I really wonder how to use ICU in Koha and that this can be a benefit (or not), example for libraries in Cyrillic? Found http://bugs.koha-community.org/bugzilla3/show_bug.cgi?id 3216 but as it applied to have already installed system?
I have some more info. Brendan did some more testing and found that this bug seems to affect libraries both with and without ICU enabled. BUT it only happens when you have the query autotruncate sys pref set to "only if the * is added."
Hello I confirm the bug with ICU and QueryStemming activated. Very annoying... M. Saby Rennes 2 University
Created attachment 14113 [details] [review] Proposed patch I've analyzed Zebra server logs and tested with yaz-client. The bug appears when a single double quote exists in a complexe search query. This appears when syspref "QueryWeightFields" is on. I found that Bug 5651 was nearly the same problem. So I added a replacement of double quotes at the same place.
The comment on the new line needs to be fixed. Also, if you can provide a detailed plan to reproduce the bug, that would be great. I've so far failed to reproduce it. I have ICU enabled and all the other sysprefs set up, but no error.
Created attachment 14144 [details] [review] Proposed patch (revisited) Oups, correct about comment. I changed correction : double quotes can be escaped : replace " by \". Test plan : ----------- - Set QueryWeightFields off - perform a simple search with a single double quote that has results. For exemple : "Paris => you get results - Set QueryWeightFields on - perform same search => you get no results CCL parser automaticaly uses double quotes around search term that is wy a single souble quote brakes parsing.
Thank you Fridolyn Your patch improves Koha's behavior, but I am not sure I can sign off. Double quotes don't break searchs anymore, but they don't force Koha to consider the query as an "exact expression", as I think they should do. Ex : One book has this title : La mémoire ou l'oubli and this autor : Alain Bosquet After applying yout patch, if I search "La mémoire ou l'oubli", with double quotes, I find the book. But if I search "L'oubli ou la mémoire" I find it too... And I can also find the book with "mémoire Bosquet" within double quotes. M. Saby Rennes 2 University
(In reply to comment #8) > Thank you Fridolyn > Your patch improves Koha's behavior, but I am not sure I can sign off. > Double quotes don't break searchs anymore, but they don't force Koha to > consider the query as an "exact expression", as I think they should do. Ok, but this is not the same problem. It can be asked as enhancement in a new bug.
Maybe it could be seen as an other problem, but I am not sure whether it be an enhancement or a correction : with some settings, double quotes are forcing Koha to treat search terms as an exact phrase, but with other settings it is not the case. And I could not information regarding the use of quotes in the manual, so I don't know what Koha is "supposed" to do... M. Saby
(In reply to comment #10) CCL syntaxe for an exact match is 'exact' after the index ; it means phrase structure and complete field. The use of double quotes for exact match is a Google search engine behavior. I can be very usefull, but maybe hard to manage in actual C4::Search.
(In reply to comment #11) > (In reply to comment #10) > CCL syntaxe for an exact match is 'exact' after the index ; it means phrase > structure and complete field. > The use of double quotes for exact match is a Google search engine behavior. > I can be very usefull, but maybe hard to manage in actual C4::Search. Very hard (read: impossible) if we want to keep using CCL2RPN. Fortunately, there's a better way! Bug 9239 introduces a new query parser that uses quotes for phrases.
OK. So I sign this bug off (and correct a typo in comments : brakes -> breaks)
Created attachment 14187 [details] [review] [PATCH] Bug 7518: searches with quotation marks don't work
QA comment: I would welcome an improved handling of quotes. But look at this simple test: Searching for "vermeer" did work without this patch but now gives: Can't call method "sort" on an undefined value t /usr/share/koha/testclone/C4/Search.pm line 407. From the zebra log: WARNING: query problem with kw,wrdl= \\"vermeer\\" ZOOM error 10014 "CCL parsing error" from diag-set 'ZOOM' at /usr/share/koha/testclone/C4/Search.pm line 359 Any chance of different results due to different Zebra config under MARC21/UNIMARC? Failed QA
Created attachment 15936 [details] [review] Proposed patch Rewritten patch. Problem comes only from C4::Seach::_build_weighted_query, depending on QueryWeightFields syspref. This code builds a complex CCL query using " around search words. So if a search word contains a ", the CCL query is wrong and you get no results. This patch simply replaces " by a space in _build_weighted_query. See commit comment.
Test plan : - Set QueryWeightFields off - Perform a serch on two words where you have results, like : centre "ville => you get results - Set QueryWeightFields on - Perform same serch => you get the same results Patch applies
Patch tested with a sandbox, by Leila <koha.aixmarseille@gmail.com>
Created attachment 16238 [details] [review] Bug 7518: searches with quotation marks don't work A search term without quotation marks returns the expected results. The same search term with quotation marks returns no results. Koha should ignore the quotation marks and return results anyway. This appears when QueryWeightFields syspref is activated. This feature composes a complex CCL query using double quotes around search words. This patch simply replaces double quotes in search words by a space. Test plan : - Set QueryWeightFields off - Perform a serch on two words where you have results, like : centre "ville => you get results - Set QueryWeightFields on - Perform same serch => you get the same results Signed-off-by: Leila <koha.aixmarseille@gmail.com>
The change looks safe, but I can't see any problems with quotation marks in master. Maybe I didn't get the configuration right?
Do you have ICU enabled? (In reply to comment #20) > The change looks safe, but I can't see any problems with quotation marks in > master. Maybe I didn't get the configuration right?
I now tested this with and without ICU and both times, I couldn't find any problems. The change doesn't look big, but I am a bit puzzled about not being able to reproduce the problem. Maybe another patch already fixed it? Please retest on current master.
Created attachment 17150 [details] screenshot of search configuration used for testing
Hello The problem described in this bug seems to be fixed in master. (But it is still present in our preproduction Koha, rebased on 10th of March) UNIMARC db, ICU QueryWeightFields Enable Queryparser Don'try to use Search "prix du blé" => 1 result (that's correct) Request : 22:55:02-02/04 zebrasrv(4) [request] Search biblios OK 1 1 1+0 RPN @attrset Bib-1 @attr 1=1016 @attr 4=6 @attr 5=1 "prix du blé" M. Saby Rennes 2 university
(In reply to comment #24) > Search "prix du blé" => 1 result (that's correct) Try with a quote into the search terms : prix du "blé
(In reply to comment #23) > Created attachment 17150 [details] > screenshot of search configuration used for testing Try without QueryAutoTruncate.
I can't try without - witout search doesn't work at all with ICU :) See bug 9976
I tested "prix du blé", "prix du "blé", prix du "blé, "blé du prix", "blé du "prix", blé du "prix" => everything is working in master, without your patch (same number of results) Mathieu
I could imagine this is a side effect of the work on the query parser. Maybe we should test if a patch is needed for 3.10.x still?
(In reply to comment #29) > I could imagine this is a side effect of the work on the query parser. Maybe > we should test if a patch is needed for 3.10.x still? Yes I think so.
Resetting version.
This is still a bug in current master : - with or without ICU - with or without UseQueryParser syspref - only with QueryWeightFields enabled
That's puzzling... Some months ago I was sure it was fixed. So you think a patch is still needed ? Mathieu
> So you think a patch is still needed ? Yes. Looking at screenshot of configuration, I think the bug appears only with QueryStemming off. In fact, to test : disable all query sysprefs unless QueryWeightFields and set QueryAutoTruncate to "only if * is added". I set back to signed off. A second signoff is welcomed.
Created attachment 19596 [details] [review] Bug 7518: searches with quotation marks don't work A search term without quotation marks returns the expected results. The same search term with quotation marks returns no results. Koha should ignore the quotation marks and return results anyway. This appears when QueryWeightFields syspref is activated. This feature composes a complex CCL query using double quotes around search words. This patch simply replaces double quotes in search words by a space. Test plan : - Set QueryWeightFields off - Perform a serch on two words where you have results, like : centre "ville => you get results - Set QueryWeightFields on - Perform same serch => you get the same results Signed-off-by: Leila <koha.aixmarseille@gmail.com> Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
I'm having difficulty reproducing this bug. For those who can, perchance are there any ZOOM error messages associated with it showing up in the Apache error log?
(In reply to Galen Charlton from comment #36) > I'm having difficulty reproducing this bug. For those who can, perchance > are there any ZOOM error messages associated with it showing up in the > Apache error log? The best way is to look at Zebra server output. - launch Zebra server manually : zebrasrv -f /home/koha/etc/koha-conf.xml - Set syspref QueryWeightFields on - Perform a search on two words. For example : centre ville : - Look at zebrasrv log : 19:15:18-31/07 zebrasrv(4) [request] Search biblios OK 0 1 1+0 RPN @attrset Bib-1 @or @or @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 "centre ville" @attr 1=4 @attr 4=1 @attr 6=3 @attr 9=28 @attr 2=102 "centre ville" @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 "centre ville" @attr 4=6 @attr 9=14 @attr 2=102 "centre ville" - Perform a search with a double quote. For example : centre "ville : - Look at zebrasrv log : 19:15:13-31/07 zebrasrv(3) [request] Search biblios OK 0 1 1+0 RPN @attrset Bib-1 @or @attr 1=36 @attr 4=1 @attr 6=3 @attr 9=32 @attr 2=102 "centre ville or ti,ext,r2=centre ville" @attr 1=36 @attr 4=1 @attr 9=26 @attr 2=102 "centre ville or wrdl,r9=centre ville" => You see that with double quote, there is some unconverted CCL into PQF query : or wrdl,r9=
Can it be pushed in master ?
Pushed to master, along with a regression test. Thanks, Fridolin!
Patch pushed to 3.14.x, will be in 3.14.3.
This patch has been pushed to 3.12.x, will be in 3.12.11. Thanks Fridolin!
(In reply to Jared Camins-Esakov from comment #12) > (In reply to comment #11) > > (In reply to comment #10) > > CCL syntaxe for an exact match is 'exact' after the index ; it means phrase > > structure and complete field. > > The use of double quotes for exact match is a Google search engine behavior. > > I can be very usefull, but maybe hard to manage in actual C4::Search. > > Very hard (read: impossible) if we want to keep using CCL2RPN. Fortunately, > there's a better way! Bug 9239 introduces a new query parser that uses > quotes for phrases. As Jared has mentioned, I think the use of double quotes to signify a phrase/exact search would also be difficult to do while using CCL2RPN. However, as some people have mentioned, it seems that Koha has had this ability in the past... This is, in part, because the default Koha search is a kw,wrdl search (i.e. a Word list search on the Keyword index). "The structure attribute values Word list (6) is supported, and maps to the boolean AND combination of words supplied. The word list is useful when Google-like bag-of-word queries need to be translated from a GUI query language to PQF." Source: http://www.indexdata.com/zebra/doc/querymodel-rpn.html In other words, doing a regular search for "professional responsibility" would not do a phrase search. Rather, it would do a search like this: "professional" AND "responsibility". In some cases, this will return the exact same results as a phrase search would, but technically speaking it's not the same because proximity and order of the words to each other doesn't take place in a "Word list" type search. The other reason why it seemed that we had this functionality in the past is probably due to conflicting system preferences which accidentally turned off QueryWeightFields. Truncation and QueryWeightFields really do not play well together. I am endeavouring to fix that. I'll open other bugs to fix that though...
It's also worth mentioning that this patch doesn't work very well when doing a regular (i.e. 'Library Catalog' or 'kw') search, when using "QueryWeightFields", and when NOT using QueryAutoTruncate. It appears to be an issue with the quoting of strings in _build_weighted_query. I won't bother sending a follow-up here, as I'm planning to refactor _build_weighted_query. NOTE: If you are using QueryAutoTruncate (ie all queries are autotruncated) then QueryWeightFields will be ignored (even though it's on by default), so you'll probably have no problems (although you also won't have any relevance searching).
All this needs to be considered in the QueryParser.