Bug 9715 - Various combinations of prefs QueryAutoTruncate, QueryFuzzy and QueryWeightFields conflict
Summary: Various combinations of prefs QueryAutoTruncate, QueryFuzzy and QueryWeightFi...
Status: NEW
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low major (vote)
Assignee: Galen Charlton
QA Contact:
URL:
Keywords:
: 9976 (view as bug list)
Depends on:
Blocks:
 
Reported: 2013-02-26 19:48 UTC by Martin Renvoize
Modified: 2020-12-03 05:58 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Martin Renvoize 2013-02-26 19:48:42 UTC
Some combinations of QueryAutoTruncate and QueryFuzzy lead to search returning 0 results for all searches.

The following combinations break search:

QueryAutoTruncate - only if * is added
QueryFuzzy - Try

and

QueryAutoTruncate - automatically.
QueryFuzzy - Don't try
Comment 1 Martin Renvoize 2013-03-15 09:47:05 UTC
Adding to this, I've done some more investigating.. it seems QueryWeightFields is also relevant.

Updated combinations that completely break search:

QueryAutoTruncate - only if * is added
QueryFuzzy - Try
QueryWeightFields - Enable

and

QueryAutoTruncate - automatically.
QueryFuzzy - Don't try
QueryWeightFields - Enable

I'm going to test this further and try to build a definitive list of what works and what doesn't.
Comment 2 Katrin Fischer 2013-04-07 07:41:07 UTC
Tested on recent master this combination is problematic:

Make sure Koha is installed to use ICU indexing.

Configuration settings:
IncludeSeeFromInSearches = Don't include
QueryAutoTruncate = automatically
QueryFuzzy = Try
QueryStemming = Try
QueryWeightFields = Enable
UseIcu = Using
UseQueryParser = Do not try

With this settings, search will work.

Now switch QueryAutoTruncate = only when * is added

Search won't work at all now.
Comment 3 Katrin Fischer 2013-04-07 07:41:27 UTC
*** Bug 9976 has been marked as a duplicate of this bug. ***
Comment 4 Marcel de Rooy 2016-01-25 13:23:17 UTC
What about Fuzzy=Try and an older Zebra 2.0.44 (on Wheezy -- still supported)?

Fuzzy adds 5=103 to the query and that results in an ERROR 120 Unsupported Truncation attribute.
BTW Note that Fuzzy is enabled by default in a new Koha (sysprefs.sql).
Verified on current master.

Not really a welcome to Koha?
Comment 5 Martin Renvoize 2018-05-16 09:22:05 UTC
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
|         Setting Combination                            |                                              Search Terms                                              | Number of Results |                                                                                                                    Notes                                                                                                                             |
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
| QueryAutoTruncate - Automatically     | frogs toads                                                                                               | 8                                     |                                                                                                                                                                                                                                                            |
| QueryFuzzy - Try/Don't try                       | frogs and toads                                                                                       | 7                                     | The missing record here contains 'toadstools' - truncation not working correctly?                                                                                       |
| QueryStemming - Try/Don't try              | frogs or toads                                                                                          | 29                                  | 'And' and 'or' treated as search operators.                                                                                                                                                                       |
|                                                                                | Adv. Search: frogs and toads                                                            | 8                                    | Apostrophe search not working                                                                                                                                                                                           |
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
| QueryAutoTruncate - only if * added    | frogs toads                                                                                               | 7                                     |                                                                                                                                                                                                                                                            |
| QueryFuzzy - don't try                                 | frogs and toads                                                                                      | 5                                    | And is treated as a search word                                                                                                                                                                                             |
| QueryStemming - don't try                       | frogs or toads                                                                                          | 0                                    |                                                                                                                                                                                                                                                             |
|                                                                               | Adv. Search: frogs and toads                                                             | 7                                    | Apostrophe search not working                                                                                                                                                                                            |
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
| QueryAutoTruncate - only if *added     | frogs toads                                                                                              | 10                                    | Includes records with  'frog' 'toad' - querystemming working as expected?                                                                                                      |
| QueryFuzzy - don't try                                | frogs and toads                                                                                      | 6                                      | And is treated as a search word                                                                                                                                                                                            |
| QueryStemming - Try                                 | frogs or toads                                                                                         | 2                                      | Or is treated as a search word (Orchid, Orangutan)                                                                                                                                                      |
|                                                                               | Adv. Search: frogs and toads                                                            | 10                                    | Apostrophe searching works (dr spock's).  Or is the s or 's just being stripped out by Query Stemming?                                             |
|                                                                               |                                                                                                                       |                                          | This seems to be the best combination - apostrophe searching and query stemming appear to be working correctly,                  |
|                                                                               |                                                                                                                       |                                          | BUT boolean operators used incorrectly.                                                                                                                                                                         |
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
| QueryAutoTruncate - only if *added     | frogs toads                                                                                               | 0                                     | No results for any search                                                                                                                                                                                                         |
| QueryFuzzy - Try                                           | frogs and toads                                                                                      | 0                                     |                                                                                                                                                                                                                                                            |
| QueryStemming - don't try/Try               | frogs or toads                                                                                         | 0                                     |                                                                                                                                                                                                                                                            |
|                                                                               | Adv. Search: frogs and toads                                                            | 0                                     |                                                                                                                                                                                                                                                            |
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
|                                                                                |                                                                                                                      |                                         | Queryfuzzy can't be used with ICU in use?                                                                                                                                                                       |
+-------------------------------------+--------------------------------------------------------+-------------------+------------------------------------------------------------------------------------------------------------------------+
Comment 6 Katrin Fischer 2018-05-16 09:24:41 UTC
Hi Martin, can you post your results elsewhere (wiki?), not easy to read here.
Comment 7 Pablo AB 2020-07-10 20:24:59 UTC
## Setting Combination

QueryAutoTruncate - Automatically
QueryFuzzy - Try/Don't try
QueryStemming - Try/Don't try

[Search Terms : Number of Results : Notes]
frogs toads : 8 :
frogs and toads : 7 : The missing record here contains 'toadstools' - truncation not working correctly?
frogs or toads : 29 : 'And' and 'or' treated as search operators.
Adv. Search: frogs and toads : 8 : Apostrophe search not working

QueryAutoTruncate - only if * added
QueryFuzzy - don't try
QueryStemming - don't try

frogs toads : 7 :
frogs and toads : 5 : And is treated as a search word
frogs or toads : 0 :
Adv. Search: frogs and toads : 7 : Apostrophe search not working 

QueryAutoTruncate - only if *added
QueryFuzzy - don't try
QueryStemming - Try

frogs toads : 0 : No results for any search
frogs and toads : 0 :
frogs or toads : 0 :
Adv. Search: frogs and toads : 0 :

QueryAutoTruncate - only if *added
QueryFuzzy - Try
QueryStemming - don't try/Try

frogs toads : 0 : No results for any search 
frogs and toads : 0 :
frogs or toads : 0 :
Adv. Search: frogs and toads : 0 :

Queryfuzzy can't be used with ICU in use?
Comment 8 Katrin Fischer 2020-07-12 11:18:53 UTC
Hi Pablo, thx for the systematic testing!

(In reply to Pablo AB from comment #7)
> ## Setting Combination
> 
> QueryAutoTruncate - Automatically
> QueryFuzzy - Try/Don't try
> QueryStemming - Try/Don't try
> 
> [Search Terms : Number of Results : Notes]
> frogs toads : 8 :
> frogs and toads : 7 : The missing record here contains 'toadstools' -
> truncation not working correctly?

I think this is due to it not containing frogs. The and is (as far as I know) interpreted as boolean operator.

> frogs or toads : 29 : 'And' and 'or' treated as search operators.
> Adv. Search: frogs and toads : 8 : Apostrophe search not working

What do you mean by Apostrophe search?

> QueryAutoTruncate - only if * added
> QueryFuzzy - don't try
> QueryStemming - don't try
> 
> frogs toads : 7 :
> frogs and toads : 5 : And is treated as a search word
> frogs or toads : 0 :
> Adv. Search: frogs and toads : 7 : Apostrophe search not working 
> 
> QueryAutoTruncate - only if *added
> QueryFuzzy - don't try
> QueryStemming - Try
> 
> frogs toads : 0 : No results for any search
> frogs and toads : 0 :
> frogs or toads : 0 :
> Adv. Search: frogs and toads : 0 :

It's hard to tell without seeing the records what's going on there and if it's correct or not. 

> QueryAutoTruncate - only if *added
> QueryFuzzy - Try
> QueryStemming - don't try/Try
> 
> frogs toads : 0 : No results for any search 
> frogs and toads : 0 :
> frogs or toads : 0 :
> Adv. Search: frogs and toads : 0 :
> 
> Queryfuzzy can't be used with ICU in use?

I think that is possible, we have had to play around with settings when we started using ICU in the first library. ,ext (exact field content) search is also not working with ICU as far as I know.
Comment 9 Pablo AB 2020-07-12 23:30:22 UTC
I just formatted Martin Renvoize's comment 5.
Comment 10 Katrin Fischer 2020-07-13 21:05:16 UTC
(In reply to Pablo AB from comment #9)
> I just formatted Martin Renvoize's comment 5.

Sorry Pablo, totally missed that!
Comment 11 David Cook 2020-12-03 05:56:14 UTC
We could just not use QueryFuzzy when UseICU is enabled and then add a warning to the syspref description?
Comment 12 David Cook 2020-12-03 05:58:34 UTC
I would note that QueryStemming, based on my tests on Bug 25900, is also problematic.