Bug 11158 - Authorities 'starts with' search returns the same as 'contains' when using ICU
Summary: Authorities 'starts with' search returns the same as 'contains' when using ICU
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Colin Campbell
QA Contact: Testopia
URL:
Keywords:
Depends on: 27299
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-29 12:40 UTC by Colin Campbell
Modified: 2021-06-07 02:41 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Text to go in the release notes:
Version(s) released in:


Attachments
Proposed patch (1.35 KB, patch)
2013-10-29 12:55 UTC, Colin Campbell
Details | Diff | Splinter Review
Bug 11158: Add complete field to authority "starts with" search (1.66 KB, patch)
2021-01-05 00:26 UTC, David Cook
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Colin Campbell 2013-10-29 12:40:16 UTC
If set up to use icu tokenisation
When doing an authorities search and specifying 'starts with' the result set returned is the same as with the option 'contains'

For example a search on starts_with 'Smith' returns

Campbell-Smith, Duncan.
Smith, Bessie, 1894-1937.
Smith, Sydney Goodsir, 1915-1975.
Smith, Victoria. 

The problem appears be that the starts-with definition is being applied to each token within the heading rather than to the heading as complete field

(A search for O hoping to retrieve O'Brian will return much more )

the correct return would be

Smith, Bessie, 1894-1937.
Smith, Sydney Goodsir, 1915-1975.
Smith, Victoria.
Comment 1 Colin Campbell 2013-10-29 12:55:13 UTC
Created attachment 22517 [details] [review]
Proposed patch

This change appears to fix the problem. At the same time it does not appear to break the behaviour with 'chr' To test on a system set up to use ICU find a test case similar to that in the bug description. Apply the patch and redo the 'starts with' search.
Comment 2 Colin Campbell 2013-10-29 13:00:11 UTC
There is also a question over the rule being used to tokenise phrase searches and what would be the best default configuration (including no rule) further investigation is dependent on bug 10729 as phrase and keyword searching have different requirements in this respect
Comment 3 Zeno Tajoli 2013-11-18 09:09:11 UTC
Hi,

I have test it on a system with ICU and with the patch of  bug 10729.
The result is this error on Opac:

ZOOM error 20003 &quot;can't set prefix query&quot; (addinfo: "@or @attr 7=1 @attr 1=Heading 0  @and  @attr 1=authtype @attr 5=100 TOPIC_TERM @attr 1=Heading  \@attr 3=1 \@attr 6=3 \@attr 4=1 \@attr 5=1 "Biblia"") from diag-set 'ZOOM'</pre>

Same error in Intranet auth search.

Do you have a proposal for use together this patch and bug 10729 patch ?
Comment 4 Colin Campbell 2013-11-21 10:36:25 UTC
I've managed to reproduce Zeno's result and I'm investigating. The problem seems to be occuring irrespective to whether or not bug 10729 is applied, (which is logical looking at the error return)
Comment 5 Colin Campbell 2013-11-21 10:54:04 UTC
Bug 5615 may have relevance here
Comment 6 Katrin Fischer 2015-02-22 09:01:18 UTC
The search part of the manual states that a 'begins with' search is not possible with ICU - this here seems a bit related. Is there way we can make this work?
Comment 7 David Cook 2021-01-04 00:01:51 UTC
(In reply to Colin Campbell from comment #2)
> There is also a question over the rule being used to tokenise phrase
> searches and what would be the best default configuration (including no
> rule) further investigation is dependent on bug 10729 as phrase and keyword
> searching have different requirements in this respect

I cover this more at https://github.com/indexdata/idzebra/issues/24#issuecomment-751260285 but yeah the tokenization is definitely the issue
Comment 8 David Cook 2021-01-04 00:27:36 UTC
I hadn't thought of the complete field attribute and truncation... interesting.

For what it's worth, without the @attr 6=3, it uses the "w" register: https://software.indexdata.com/zebra/doc/querymodel-zebra.html#querymodel-pqf-apt-mapping-structuretype
Comment 9 David Cook 2021-01-05 00:15:03 UTC
Due to my work on Bug 27299, I have renewed interest in this one.

In koha-testing-docker, the following URL http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl?op=do_search&type=intranet&authtypecode=&marclist=mainmainentry&and_or=and&excluding=&operator=start&value=Espen&orderby=HeadingAsc will generate the following Zebra query:

@attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 5=1 @attr 4=1 @attr 3=2 @attr 1=Heading-Main Espen

The key part of that is the following:

@attr 5=1 @attr 4=1 @attr 3=2 @attr 1=Heading-Main Espen

Breaking that down:

@attr 5=1 -> Right truncation
@attr 4=1 -> Phrase structure
@attr 3=2 -> Position first in subfield
@attr 1=Heading-Main Espen

As Colin notes, the key is really the addition of a @attr 6=3 to force a complete field search, which uses the "p" register. 

In my testing, we can leave @attr 3=2 as is, since it doesn't really make a difference. But adding @attr 6=3 fixes the issue - at least when the patch from Bug 27299 is applied...
Comment 10 David Cook 2021-01-05 00:21:08 UTC
Colin's patch doesn't apply anymore, so I'm going to do a new one, which just adds @attr 6=3.

I'm also adding a dependency on Bug 27299 since that'll be crucial
Comment 11 David Cook 2021-01-05 00:26:21 UTC
Created attachment 114829 [details] [review]
Bug 11158: Add complete field to authority "starts with" search

This patch adds "complete field" to the authority "starts with"
search so that it uses the untokenized "p" register.

Test plan:
1. Apply the patch
2. koha-plack --restart kohadev
3. Go to http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl
4. Type in "Espen" into the search box and hit "Submit"
5. Note that there are 3 results
6. Change "contains" to "starts with" and hit "Submit"
7. Note that no results are returned
8. Change the search from "Espen" to "Sandberg" and hit "Submit"
9. Note that 3 results are returned
10. Experiment to your heart's content and rejoice at your new found power
Comment 12 Katrin Fischer 2021-06-04 10:12:21 UTC
The test plan works ok on current master for me, but I assume this is because the kohadevbox is using ICU by default. I am not sure how to activate ICU there.
Comment 13 David Cook 2021-06-07 02:41:08 UTC
(In reply to Katrin Fischer from comment #12)
> The test plan works ok on current master for me, but I assume this is
> because the kohadevbox is using ICU by default. I am not sure how to
> activate ICU there.

I think you mean using CHR by default. I wonder if we should change to ICU default some time. 

It involves updating /etc/koha/zebradb/etc/default.idx and then re-indexing Zebra.

Although I didn't have that in my test plan I wonder why...