Bug 11158 - Authorities 'starts with' search returns the same as 'contains' when using ICU
Summary: Authorities 'starts with' search returns the same as 'contains' when using ICU
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal (vote)
Assignee: David Cook
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on: 27299
Blocks:
  Show dependency treegraph
 
Reported: 2013-10-29 12:40 UTC by Colin Campbell
Modified: 2023-12-28 20:42 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
This fixes searching authorities when using ICU* so that 'starts with' searching works correctly. Before this, a 'starts with' search would return the same results as 'contains'. Technical details: this adds the "complete field" to the authority "starts with" search so that it uses the untokenized "p" register. (* ICU is a feature of the Zebra search engine that can be configured to make searching with non-latin languages (such as Chinese and Arabic) work correctly.)
Version(s) released in:
22.11.00


Attachments
Proposed patch (1.35 KB, patch)
2013-10-29 12:55 UTC, Colin Campbell
Details | Diff | Splinter Review
Bug 11158: Add complete field to authority "starts with" search (1.66 KB, patch)
2021-01-05 00:26 UTC, David Cook
Details | Diff | Splinter Review
Bug 11158: Add complete field to authority "starts with" search (1.71 KB, patch)
2022-07-03 19:20 UTC, David Nind
Details | Diff | Splinter Review
Bug 11158: Add complete field to authority "starts with" search (1.77 KB, patch)
2022-07-14 09:31 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Colin Campbell 2013-10-29 12:40:16 UTC
If set up to use icu tokenisation
When doing an authorities search and specifying 'starts with' the result set returned is the same as with the option 'contains'

For example a search on starts_with 'Smith' returns

Campbell-Smith, Duncan.
Smith, Bessie, 1894-1937.
Smith, Sydney Goodsir, 1915-1975.
Smith, Victoria. 

The problem appears be that the starts-with definition is being applied to each token within the heading rather than to the heading as complete field

(A search for O hoping to retrieve O'Brian will return much more )

the correct return would be

Smith, Bessie, 1894-1937.
Smith, Sydney Goodsir, 1915-1975.
Smith, Victoria.
Comment 1 Colin Campbell 2013-10-29 12:55:13 UTC
Created attachment 22517 [details] [review]
Proposed patch

This change appears to fix the problem. At the same time it does not appear to break the behaviour with 'chr' To test on a system set up to use ICU find a test case similar to that in the bug description. Apply the patch and redo the 'starts with' search.
Comment 2 Colin Campbell 2013-10-29 13:00:11 UTC
There is also a question over the rule being used to tokenise phrase searches and what would be the best default configuration (including no rule) further investigation is dependent on bug 10729 as phrase and keyword searching have different requirements in this respect
Comment 3 Zeno Tajoli 2013-11-18 09:09:11 UTC
Hi,

I have test it on a system with ICU and with the patch of  bug 10729.
The result is this error on Opac:

ZOOM error 20003 &quot;can't set prefix query&quot; (addinfo: "@or @attr 7=1 @attr 1=Heading 0  @and  @attr 1=authtype @attr 5=100 TOPIC_TERM @attr 1=Heading  \@attr 3=1 \@attr 6=3 \@attr 4=1 \@attr 5=1 "Biblia"") from diag-set 'ZOOM'</pre>

Same error in Intranet auth search.

Do you have a proposal for use together this patch and bug 10729 patch ?
Comment 4 Colin Campbell 2013-11-21 10:36:25 UTC
I've managed to reproduce Zeno's result and I'm investigating. The problem seems to be occuring irrespective to whether or not bug 10729 is applied, (which is logical looking at the error return)
Comment 5 Colin Campbell 2013-11-21 10:54:04 UTC
Bug 5615 may have relevance here
Comment 6 Katrin Fischer 2015-02-22 09:01:18 UTC
The search part of the manual states that a 'begins with' search is not possible with ICU - this here seems a bit related. Is there way we can make this work?
Comment 7 David Cook 2021-01-04 00:01:51 UTC
(In reply to Colin Campbell from comment #2)
> There is also a question over the rule being used to tokenise phrase
> searches and what would be the best default configuration (including no
> rule) further investigation is dependent on bug 10729 as phrase and keyword
> searching have different requirements in this respect

I cover this more at https://github.com/indexdata/idzebra/issues/24#issuecomment-751260285 but yeah the tokenization is definitely the issue
Comment 8 David Cook 2021-01-04 00:27:36 UTC
I hadn't thought of the complete field attribute and truncation... interesting.

For what it's worth, without the @attr 6=3, it uses the "w" register: https://software.indexdata.com/zebra/doc/querymodel-zebra.html#querymodel-pqf-apt-mapping-structuretype
Comment 9 David Cook 2021-01-05 00:15:03 UTC
Due to my work on Bug 27299, I have renewed interest in this one.

In koha-testing-docker, the following URL http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl?op=do_search&type=intranet&authtypecode=&marclist=mainmainentry&and_or=and&excluding=&operator=start&value=Espen&orderby=HeadingAsc will generate the following Zebra query:

@attrset Bib-1 @or @attr 1=Heading @attr 7=1 0 @attr 5=1 @attr 4=1 @attr 3=2 @attr 1=Heading-Main Espen

The key part of that is the following:

@attr 5=1 @attr 4=1 @attr 3=2 @attr 1=Heading-Main Espen

Breaking that down:

@attr 5=1 -> Right truncation
@attr 4=1 -> Phrase structure
@attr 3=2 -> Position first in subfield
@attr 1=Heading-Main Espen

As Colin notes, the key is really the addition of a @attr 6=3 to force a complete field search, which uses the "p" register. 

In my testing, we can leave @attr 3=2 as is, since it doesn't really make a difference. But adding @attr 6=3 fixes the issue - at least when the patch from Bug 27299 is applied...
Comment 10 David Cook 2021-01-05 00:21:08 UTC
Colin's patch doesn't apply anymore, so I'm going to do a new one, which just adds @attr 6=3.

I'm also adding a dependency on Bug 27299 since that'll be crucial
Comment 11 David Cook 2021-01-05 00:26:21 UTC
Created attachment 114829 [details] [review]
Bug 11158: Add complete field to authority "starts with" search

This patch adds "complete field" to the authority "starts with"
search so that it uses the untokenized "p" register.

Test plan:
1. Apply the patch
2. koha-plack --restart kohadev
3. Go to http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl
4. Type in "Espen" into the search box and hit "Submit"
5. Note that there are 3 results
6. Change "contains" to "starts with" and hit "Submit"
7. Note that no results are returned
8. Change the search from "Espen" to "Sandberg" and hit "Submit"
9. Note that 3 results are returned
10. Experiment to your heart's content and rejoice at your new found power
Comment 12 Katrin Fischer 2021-06-04 10:12:21 UTC
The test plan works ok on current master for me, but I assume this is because the kohadevbox is using ICU by default. I am not sure how to activate ICU there.
Comment 13 David Cook 2021-06-07 02:41:08 UTC
(In reply to Katrin Fischer from comment #12)
> The test plan works ok on current master for me, but I assume this is
> because the kohadevbox is using ICU by default. I am not sure how to
> activate ICU there.

I think you mean using CHR by default. I wonder if we should change to ICU default some time. 

It involves updating /etc/koha/zebradb/etc/default.idx and then re-indexing Zebra.

Although I didn't have that in my test plan I wonder why...
Comment 14 David Cook 2022-06-27 00:34:46 UTC
Noticed this as one of the "10 Oldest bugs" under "Needs Signoff". 

Figured I'd update to the Assignee to me too since I apparently wrote the patch 1.5 years ago...
Comment 15 David Nind 2022-07-03 19:20:12 UTC
Created attachment 137049 [details] [review]
Bug 11158: Add complete field to authority "starts with" search

This patch adds "complete field" to the authority "starts with"
search so that it uses the untokenized "p" register.

Test plan:
1. Apply the patch
2. koha-plack --restart kohadev
3. Go to http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl
4. Type in "Espen" into the search box and hit "Submit"
5. Note that there are 3 results
6. Change "contains" to "starts with" and hit "Submit"
7. Note that no results are returned
8. Change the search from "Espen" to "Sandberg" and hit "Submit"
9. Note that 3 results are returned
10. Experiment to your heart's content and rejoice at your new found power

Signed-off-by: David Nind <david@davidnind.com>
Comment 16 David Nind 2022-07-03 19:28:06 UTC
Testing notes (using koha-testing-docker):

1. This is my first time setting up ICU - I followed the instructions at https://wiki.koha-community.org/wiki/ICU_chains_configuration except for enabling system preference UseICU, as it no longer exists

2. Queries about setting up ICU:
   . Are the instructions still current?
   . There is no system preference UseICU, but there is UseICUStyleQuotes
   . When editing /etc/koha/zebradb/etc/default.idx I noticed the URX (URL) index, Numeric index, and Sort register sections reference *.chr files - do these need changing as well?

3. I used the query in the description (Smith) as well when testing.
Comment 17 Martin Renvoize 2022-07-14 09:31:08 UTC
Created attachment 137699 [details] [review]
Bug 11158: Add complete field to authority "starts with" search

This patch adds "complete field" to the authority "starts with"
search so that it uses the untokenized "p" register.

Test plan:
1. Apply the patch
2. koha-plack --restart kohadev
3. Go to http://localhost:8081/cgi-bin/koha/authorities/authorities-home.pl
4. Type in "Espen" into the search box and hit "Submit"
5. Note that there are 3 results
6. Change "contains" to "starts with" and hit "Submit"
7. Note that no results are returned
8. Change the search from "Espen" to "Sandberg" and hit "Submit"
9. Note that 3 results are returned
10. Experiment to your heart's content and rejoice at your new found power

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 18 Martin Renvoize 2022-07-14 09:31:53 UTC
Nice one David and David!..

It's great to see Zebra having a little love still..

All works as described, so great.. Passing QA
Comment 19 David Cook 2022-07-18 00:18:03 UTC
(In reply to Martin Renvoize from comment #18)
> Nice one David and David!..
> 
> It's great to see Zebra having a little love still..

I'll keep showing it the love until all my libraries jump ship to Elasticsearch!
Comment 20 Tomás Cohen Arazi 2022-07-18 13:31:10 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!