Bug 13414 - Enable right truncation with search by relevance
Summary: Enable right truncation with search by relevance
Status: RESOLVED DUPLICATE of bug 12430
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: Koha Team University Lyon 3
QA Contact: Marcel de Rooy
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2014-12-09 09:59 UTC by Koha Team University Lyon 3
Modified: 2023-07-28 20:11 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Enable right truncation with search by relevance (5.18 KB, patch)
2014-12-09 09:59 UTC, Koha Team University Lyon 3
Details | Diff | Splinter Review
Bug 13414 - Enable automatic right truncation while searching by relevance (5.22 KB, patch)
2014-12-10 19:58 UTC, Koha Team University Lyon 3
Details | Diff | Splinter Review
Bug 13414 - Enable automatic right truncation while searching by relevance (5.25 KB, patch)
2014-12-11 12:56 UTC, Biblibre Sandboxes
Details | Diff | Splinter Review
Bug 13414 - Enable automatic right truncation while searching by relevance (5.44 KB, patch)
2015-04-17 10:32 UTC, Marcel de Rooy
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Koha Team University Lyon 3 2014-12-09 09:59:49 UTC Comment hidden (obsolete)
Comment 1 Katrin Fischer 2014-12-09 20:51:16 UTC
I am a bit worried about the database update - wouldn't that set your syspref to 1 even if the it was 0 before? Maybe just changing the values that need to be changed would work?
Comment 2 Koha Team University Lyon 3 2014-12-10 19:58:47 UTC Comment hidden (obsolete)
Comment 3 Koha Team University Lyon 3 2014-12-11 09:35:50 UTC
Well, painfully start for this patch, my assertion about manual truncation was wrong. So consider this comment cancel the previous one (comment 2) :

Moreover some more precisions were missing :
By now, relevance seems to not work at all when Automatic truncation is activated or manual Truncation is used (* wilcard ) : the results are sorted by biblionumber ascending.
It's automatic right truncation combined with relevance that this patch brings. 
I just add right truncation as a trial because it seems to match the essential and common needs : getting the plural and very closed terms. I fear going further would be a bit of tuning.  
By caution, I add it as a third option but maybe, it could replace the existing enable one.




TEST PLAN :

1) Apply patch. Set QueryWeightFields syspref on default option (relevance without truncation).

2) choose a keysearch not too much common so as to get only a small batch of results : note the number of results. 
Note that results are sorted by biblionumber ascending and not by relevance. To check, pass the cursor over the titles.That displays the link address in left down corner of active window : biblionumber is visible at the end of the address.
3) redo the search adding an s plural to your keysearch (or anything so as keeping keysearch as radical) : note the number of results. this can be higher than previous ones.
4) Among these last results, try to find a record where only the plural form is present (checking in Marc tab). Call it recordTest.
5) Turn QueryWeightFields syspref on right truncation option,
redo 1 and note the results number. It's not any more sorted by biblionumber. Little more by relevance( little more because relevance in koha is far from beeing perfect : for example high ranking on exact title doesn't make effects). There could be results where the plural form of keyword had been found (and the singular one is not present), more possibly at the end of the list.
To be sure, look for recordTest record.
6) redo 3 and note the number of results. this one should be smaller (these results must be included in previous one).
Comment 4 Biblibre Sandboxes 2014-12-11 12:56:34 UTC
Patch tested with a sandbox, by kohateam rennes 2 <koha@listes.univ-rennes2.fr>
Comment 5 Biblibre Sandboxes 2014-12-11 12:56:52 UTC Comment hidden (obsolete)
Comment 6 Martin Renvoize 2015-01-06 11:37:19 UTC
David, as the current zebra guru, any chance you could give these patches a glance over?
Comment 7 David Cook 2015-01-06 22:45:26 UTC
Comment on attachment 34316 [details] [review]
Bug 13414 - Enable automatic right truncation while searching by relevance

Review of attachment 34316 [details] [review]:
-----------------------------------------------------------------

::: C4/Search.pm
@@ +1030,5 @@
>            #$weighted_query .=" or kw,wrdl,r5=\"$operand\"";            # word list any
>          $weighted_query .= " or wrdl,fuzzy,r8=\"$operand\""
>            if $fuzzy_enabled;    # add fuzzy, word list
> +        if ( $stemming and $stemmed_operand ) {
> +            $weighted_query .= " or wrdl,right-Truncation,r9=\"$stemmed_operand\""

No functional difference... would be better to have this in a separate patch.

@@ +1620,4 @@
>                  # Apply Truncation
>                  if (
>                      scalar(@$righttruncated) + scalar(@$lefttruncated) +
> +                    scalar(@$rightlefttruncated) > 0 and $weight_fields < 2 )

This will disable all truncation except for right truncation (which will be forced), if "QueryWeightFields" is set to 2.

I don't love this, but seeing as truncation currently disables "QueryWeightFields", it might be fine.
Comment 8 David Cook 2015-01-06 23:08:25 UTC
(In reply to Martin Renvoize from comment #6)
> David, as the current zebra guru, any chance you could give these patches a
> glance over?

Honestly, "buildQuery" and "_build_weighted_query" are so badly written that any change is probably an improvement :p.

While I think this patch is a workaround, I think it's probably a reasonable enough workaround. 

--

What we really need to do is have functions that test for "QueryStemming", "QueryFuzzy", "QueryWeightFields", "QueryAutoTruncate", manual truncation, and index type THEN add the appropriate qualifiers. Rather than the bizarre mishmash that we have currently. 

Ideally, QueryParser should do this, but it hasn't been finished yet.

Personally, I want to re-write buildQuery, but haven't had the time. I'd like to use an object or a hash for each atomic query, process them (according to our sysprefs and to make them valid CCL), and then build those atomic queries into a the complete query.

It's basically what we do now, except you can't do everything to a query when realistically you should be able to do so.
Comment 9 Katrin Fischer 2015-01-18 16:08:23 UTC
David, thx for your comments. Could you give this a second sign-off?
Comment 10 David Cook 2015-01-20 02:14:25 UTC
(In reply to Katrin Fischer from comment #9)
> David, thx for your comments. Could you give this a second sign-off?

Yep, but not today. I'll put it on my to do list for later this week!
Comment 11 David Cook 2015-02-04 03:15:52 UTC
Comment on attachment 34316 [details] [review]
Bug 13414 - Enable automatic right truncation while searching by relevance

Review of attachment 34316 [details] [review]:
-----------------------------------------------------------------

::: installer/data/mysql/sysprefs.sql
@@ +343,4 @@
>  ('QueryAutoTruncate','1',NULL,'If ON, query truncation is enabled by default','YesNo'),
>  ('QueryFuzzy','1',NULL,'If ON, enables fuzzy option for searches','YesNo'),
>  ('QueryStemming','1',NULL,'If ON, enables query stemming','YesNo'),
> +('QueryWeightFields','1','0|1|2','Enable field weighting research','Choice'),

"Enable field weighting research"?

What does research mean in this context?
Comment 12 Koha Team University Lyon 3 2015-02-04 08:53:46 UTC
Yes, it would be more exact to say "Enable field weighting sorting" or to keep the initial label "enables field weighting"
Comment 13 Katrin Fischer 2015-03-16 01:06:30 UTC
David, Team Lyon3, how do we proceed here?
Would you be ok with me changing the sysprefs.sql file back to the old label?
Comment 14 Koha Team University Lyon 3 2015-03-16 09:12:41 UTC
(In reply to Katrin Fischer from comment #13)
> David, Team Lyon3, how do we proceed here?
> Would you be ok with me changing the sysprefs.sql file back to the old label?

it's ok for me
          Olivier Crouzet
Comment 15 David Cook 2015-03-22 22:34:24 UTC
(In reply to Katrin Fischer from comment #13)
> David, Team Lyon3, how do we proceed here?
> Would you be ok with me changing the sysprefs.sql file back to the old label?

It doesn't really matter too much what's in sysprefs.sql, since it won't show up in the web UI, but I figure keeping the original label is probably the best idea.
Comment 16 Marcel de Rooy 2015-04-17 10:32:22 UTC
Created attachment 38024 [details] [review]
Bug 13414 - Enable automatic right truncation while searching by relevance

This makes slight modification in build_weighted_query function in order to make
automatic right truncation possible when QueryWeightFields syspref activated (even little ranked)
Modify this syspref type from boolean to choice in order to add
this possiblity as a third option : Search by relevance with automatic right truncation.

Signed-off-by: kohateam rennes 2 <koha@listes.univ-rennes2.fr>
Comment 17 Marcel de Rooy 2015-04-17 10:32:46 UTC
Just a rebase yet..
Comment 18 Marcel de Rooy 2015-04-17 12:52:56 UTC
QA Comment:
Thanks for sending your patch. I have been thinking about your patch a while.

But in the end it seems to me that we should not be mixing the features ranking and truncation in one syspref. Search.pm is already hard to maintain. 

A crucial piece of your code: 
-        $weighted_query .= " or wrdl,r9=\"$operand\"";
-
+        } elsif ( $weight_fields == 2 ) {
+            $weighted_query .= " or wrdl,right-Truncation,r9=\"$operand\"";
+        } else {
+            $weighted_query .= " or wrdl,r9=\"$operand\"";

Question: Could you think of another way to determine when we want right truncation in the deleted first line? Do we really need an adjusted pref? 

Since this is a complicated patch (changing Search.pm), I would also like a thorough test plan. There is no test plan now.
And I would appreciate to hear from the one who signed off: What did you exactly test?

Small stuff: 
[1] Add type choice in the pref file.
[2] There is no print statement in the db rev.

Moving this to Failed QA for now.
Comment 19 Fridolin Somers 2015-10-23 13:47:50 UTC
Syspref could be more friendly with strings instead of numbers.
My 2c
Comment 20 Fridolin Somers 2016-08-30 12:41:07 UTC
How is this different from enabling "QueryStemming" ?
Automatic truncation can lead to huge number of results, consider the automatic truncation of author names and short words.
Comment 21 David Cook 2016-08-31 03:06:07 UTC
(In reply to Fridolin SOMERS from comment #20)
> How is this different from enabling "QueryStemming" ?
> Automatic truncation can lead to huge number of results, consider the
> automatic truncation of author names and short words.

That's a good question.

I suppose we'd have to look at how QueryStemming works. At a glance, it only adds right truncation if it was able to stem the operand I think.

I think Zebra takes care of manually handling "operand*" type queries...

So yeah... I don't know about automatic truncation. I think some users expect it, so they might do a query like "au:fish" and expect to get back results for "Fishburne" as well as "Fisher". 

If you had QueryStemming on, I doubt "fish" would produce a stemmed operand since it's already a stem, so that probably wouldn't work unless you had automatic right truncation.

That said, I don't know if that's a good thing... I personally think explicit truncation/wildcarding is a better idea..
Comment 22 Koha Team University Lyon 3 2016-08-31 15:00:23 UTC
(In reply to David Cook from comment #21)
> (In reply to Fridolin SOMERS from comment #20)
> > How is this different from enabling "QueryStemming" ?
> > Automatic truncation can lead to huge number of results, consider the
> > automatic truncation of author names and short words.
> 
> That's a good question.
> 
> I suppose we'd have to look at how QueryStemming works. At a glance, it only
> adds right truncation if it was able to stem the operand I think.
> 
> I think Zebra takes care of manually handling "operand*" type queries...
> 
> So yeah... I don't know about automatic truncation. I think some users
> expect it, so they might do a query like "au:fish" and expect to get back
> results for "Fishburne" as well as "Fisher". 
> 
> If you had QueryStemming on, I doubt "fish" would produce a stemmed operand
> since it's already a stem, so that probably wouldn't work unless you had
> automatic right truncation.
> 
> That said, I don't know if that's a good thing... I personally think
> explicit truncation/wildcarding is a better idea..

Seems that QueryStemming does right truncation even if AutomaticTruncation is disabled
provided that QueryWeightField is on
example in Lyon3.
searching the terms "fonction hypothese" in our OPAC (configured with AutomaticTruncation on, Querystemming and QueryWeightfield off) get 2 results :
https://catalogue.univ-lyon3.fr/cgi-bin/koha/opac-detail.pl?biblionumber=373203
https://catalogue.univ-lyon3.fr/cgi-bin/koha/opac-detail.pl?biblionumber=57190

Note that the term "fonction" exists nowhere as such in the 2 records.

In our test instance, with QueryStemming and QueryWeightField on, Autotruncate off, we get these 2 results allthesame.
But with QueryStemming on, QueryWeightField and Autotruncate off : no results.

Olivier Crouzet
Comment 23 Fridolin Somers 2016-09-09 15:13:21 UTC
(In reply to David Cook from comment #21)
> That said, I don't know if that's a good thing... I personally think
> explicit truncation/wildcarding is a better idea..
Me too.
You may customize OPAC to add a tip text to help users.
Comment 24 David Cook 2016-09-12 02:43:17 UTC
(In reply to Fridolin SOMERS from comment #23)
> Me too.
> You may customize OPAC to add a tip text to help users.

Yeah, I keep thinking of some way to provide a concise but useful set of search tips for OPAC users... something unobtrusive but easily accessible...
Comment 25 Barton Chittenden 2016-09-12 12:48:36 UTC
(In reply to David Cook from comment #24)
> (In reply to Fridolin SOMERS from comment #23)
> > Me too.
> > You may customize OPAC to add a tip text to help users.
> 
> Yeah, I keep thinking of some way to provide a concise but useful set of
> search tips for OPAC users... something unobtrusive but easily accessible...

I think that a 'search hints' link below the search bar would not be out of place.
Comment 26 David Cook 2016-09-12 23:56:25 UTC
(In reply to Barton Chittenden from comment #25)
> I think that a 'search hints' link below the search bar would not be out of
> place.

Yeah, that's kind of what I'm thinking, although the existence of the QueryParser and ElasticSearch as well as the normal default Koha searching, I don't know how a person could really provide search hints in a meaningful way. 

In a way, Koha has too many options for search...
Comment 27 Katrin Fischer 2023-07-28 20:11:44 UTC
(In reply to Koha Team University Lyon 3 from comment #3)
> Well, painfully start for this patch, my assertion about manual truncation
> was wrong. So consider this comment cancel the previous one (comment 2) :
> 
> Moreover some more precisions were missing :
> By now, relevance seems to not work at all when Automatic truncation is
> activated or manual Truncation is used (* wilcard ) : the results are sorted
> by biblionumber ascending.
> It's automatic right truncation combined with relevance that this patch
> brings.  the number of results. this one should be smaller (these
> results must be included in previous one).

I believe that we fixed this:

Bug 12430 - Relevance ranking should also be used without QueryWeightFields system preference

*** This bug has been marked as a duplicate of bug 12430 ***