Bug 26533 - Searching authorities using 'is exactly' doesn't work as expected
Summary: Searching authorities using 'is exactly' doesn't work as expected
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching (show other bugs)
Version: master
Hardware: All All
: P5 - low minor (vote)
Assignee: Aleisha Amohia
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks: 28182 28475
  Show dependency treegraph
 
Reported: 2020-09-24 21:43 UTC by Aleisha Amohia
Modified: 2021-05-28 02:37 UTC (History)
4 users (show)

See Also:
Change sponsored?: Sponsored
Patch complexity: Small patch
Text to go in the release notes:
Searching authorities using 'is exactly' was matching on any word in the heading. Now it is matching the heading exactly (the entire heading).
Version(s) released in:
21.05.00,20.11.06,20.05.12


Attachments
Bug 26533: Fix 'is exactly' search for authorities (9.23 KB, patch)
2021-03-15 20:37 UTC, Aleisha Amohia
Details | Diff | Splinter Review
Bug 26533: Fix 'is exactly' search for authorities (9.28 KB, patch)
2021-03-16 00:50 UTC, David Cook
Details | Diff | Splinter Review
Bug 26533: Fix 'is exactly' search for authorities (9.34 KB, patch)
2021-04-20 18:26 UTC, Nick Clemens
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Aleisha Amohia 2020-09-24 21:43:37 UTC
When searching authorities using the 'is exactly' limiter, it's expected that the search term needs to match the authority heading EXACTLY. However, it appears to match on any word in the authority heading.

For example, I have an authority called 'Atlantic salmon fishing Ireland'. I am able to get search results for this authority when searching 'is exactly' 'atlantic'. But obviously, the heading is not exactly atlantic. So I shouldn't get results from this.
Comment 1 Fridolin Somers 2020-12-01 14:56:14 UTC
Code tries to perform "all of subfield field must match" :
https://git.koha-community.org/Koha-community/Koha/src/commit/be9f64b638335cac7465ca302e187c7914534fbe/C4/AuthoritiesMarc.pm#L184

Zebra doc indicates 6=3 is implemented :
https://software.indexdata.com/zebra/doc/querymodel-rpn.html#querymodel-bib1-structure

But looks like it does not work with actual Zebra version.
Maybe it was implemented in GRS-1 mode.
Comment 2 Fridolin Somers 2021-03-11 08:05:59 UTC
Ohhh but its is worse than that.

In Template, authorities search "is exactly" uses operator "is" :

https://git.koha-community.org/Koha-community/Koha/src/commit/0381d9e52407fa7172ac18a85e0db8e62edc6c90/koha-tmpl/intranet-tmpl/prog/en/includes/authorities-search.inc#L90

Operator "is" searches "Phrase, No truncation" :

https://git.koha-community.org/Koha-community/Koha/src/commit/0381d9e52407fa7172ac18a85e0db8e62edc6c90/C4/AuthoritiesMarc.pm#L173

Is this a bug ?

Looks like "exact" operator is used by C4::Heading.
Comment 3 Aleisha Amohia 2021-03-15 20:37:36 UTC
Created attachment 118255 [details] [review]
Bug 26533: Fix 'is exactly' search for authorities

The exact search 6=3 is implemented, but it isn't used as expected when
you select 'is exactly' from the authorities search dropdown. This patch
remedies that.

To test:
1) Ensure you have an authority with more than one word in the name,
    i.e. Electric power production
2) Go to Authorities in the staff client.
3) Change the dropdown to 'is exactly' and search for one of the words
in your record's title, i.e. 'power' in the 'main heading' search
4) Confirm your authority shows in the results, even though it is not an
exact match.
5) Search for a biblio record and go to edit the record.
6) Go to Tab 6 and click the plugin button next to one of the 6xx$a
fields to trigger the authority finder plugin
7) Repeat Step 3 and Step 4
8) Go to the OPAC and go to Authority Search
9) Repeat Step 3 and Step 4

Apply patch

10) Repeat Steps 1 to 9, confirm that this time the record does NOT show
in search results.
11) Repeat Steps 1 to 9 but this time search for the authority record's
full title, i.e. 'Electric power production'. Confirm the record does
show in the search results because the search term now matches title exactly.

Sponsored-by: Education Services Australia SCIS
Comment 4 David Cook 2021-03-15 22:24:26 UTC
Aleisha, is this using ICU or CHR?

I know Katrin and I were talking about this sort of things recently.
Comment 5 David Cook 2021-03-15 22:25:43 UTC
Bug 27299 for ICU but if it's CHR then nevermind.

I'll make a note to come back to this too..
Comment 6 Aleisha Amohia 2021-03-15 22:26:31 UTC
(In reply to David Cook from comment #4)
> Aleisha, is this using ICU or CHR?
> 
> I know Katrin and I were talking about this sort of things recently.

This is using ICU
Comment 7 David Cook 2021-03-15 22:33:57 UTC
(In reply to Aleisha Amohia from comment #6)
> (In reply to David Cook from comment #4)
> > Aleisha, is this using ICU or CHR?
> > 
> > I know Katrin and I were talking about this sort of things recently.
> 
> This is using ICU

That would be good to include in the test plan, although I think the problem probably exists for both CHR and ICU.

Regarding Bug 27299, it has been in master for a little while, so probably not relevant here.

I'm looking at C4::AuthoritiesMarc and I'm loving how the comments for 'is' and 'exactly' are the same, but clearly they're not the same, since 'is' uses the 'w' register and 'exactly' uses the 'p' register.
Comment 8 Aleisha Amohia 2021-03-15 22:36:07 UTC
(In reply to David Cook from comment #7)
> (In reply to Aleisha Amohia from comment #6)
> > (In reply to David Cook from comment #4)
> > > Aleisha, is this using ICU or CHR?
> > > 
> > > I know Katrin and I were talking about this sort of things recently.
> > 
> > This is using ICU
> 
> That would be good to include in the test plan, although I think the problem
> probably exists for both CHR and ICU.

Yes I am pretty sure it does

> 
> Regarding Bug 27299, it has been in master for a little while, so probably
> not relevant here.

We had the patch from Bug 27299 applied and it didn't seem to solve the problem for us

> 
> I'm looking at C4::AuthoritiesMarc and I'm loving how the comments for 'is'
> and 'exactly' are the same, but clearly they're not the same, since 'is'
> uses the 'w' register and 'exactly' uses the 'p' register.

Yes a little strange! Hopefully this switch to using the 'exact' operator means we're doing the right search now.
Comment 9 David Cook 2021-03-15 22:40:09 UTC
(In reply to Aleisha Amohia from comment #8)
> > Regarding Bug 27299, it has been in master for a little while, so probably
> > not relevant here.
> 
> We had the patch from Bug 27299 applied and it didn't seem to solve the
> problem for us
> 

Yeah it wouldn't help with 'is' but would help with 'exact'.

> > 
> > I'm looking at C4::AuthoritiesMarc and I'm loving how the comments for 'is'
> > and 'exactly' are the same, but clearly they're not the same, since 'is'
> > uses the 'w' register and 'exactly' uses the 'p' register.
> 
> Yes a little strange! Hopefully this switch to using the 'exact' operator
> means we're doing the right search now.

Yeah, I think your patch is good. I'll spin up a KTD to test it in a minute. I should be doing other things but search issues always tickle me. 

I wonder if we should actually get rid of the 'is' operator all together. Not that you'd need to do that now, but I figure if we have eyes on it now it might be good to think about.
Comment 10 David Cook 2021-03-15 22:49:08 UTC
(In reply to David Cook from comment #9)
>
> I wonder if we should actually get rid of the 'is' operator all together.
> Not that you'd need to do that now, but I figure if we have eyes on it now
> it might be good to think about.

Grepping around for 'is' and I see C4/Heading.pm, C4/AuthoritiesMarc.pm, cataloguing/value_builder/unimarc_field_225a.pl, Koha/SearchEngine/Elasticsearch/QueryBuilder.pm, t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t...

Grepping for "is" finds the following:
koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/value_builder/unimarc_field_210c.tt
koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/value_builder/unimarc_field_686a.tt
koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/value_builder/unimarc_field_686a.tt
koha-tmpl/intranet-tmpl/prog/en/modules/cataloguing/value_builder/unimarc_field_686a.tt

So we can't get rid of 'is' as it looks like those UNIMARC plugins are using it at least
Comment 11 David Cook 2021-03-16 00:49:39 UTC
My test plan:

0) Start koha-testing-docker
1) Determine "Advanced language series." as suitable authority for testing
2) Go to Authorities in staff client
3) Change dropdown to "is exactly" and search "language"
4) Confirm "Advanced language series." is the first hit
5) Go to http://localhost:8081/cgi-bin/koha/cataloguing/addbiblio.pl?biblionumber=37
6) Go to field 830 and click the Tag Editor
7) Repeat steps 3 and 4
8) Go to http://localhost:8080/cgi-bin/koha/opac-authorities-home.pl
9) Repeat steps 3 and 4

Apply patch and run koha-plack --restart kohadev

10) Repeat steps 1-9 but confirm that this time the record does not show in the search results
11) Repeat steps 1-9 but this time search for Advanced language series.

--

Note that I did get 2 hits for step 10 for the OPAC's Authority search, because it's searching through the whole record and there are some general subdivisions that are exactly Language.

A bit unexpected but reasonable.
Comment 12 David Cook 2021-03-16 00:50:20 UTC
Created attachment 118257 [details] [review]
Bug 26533: Fix 'is exactly' search for authorities

The exact search 6=3 is implemented, but it isn't used as expected when
you select 'is exactly' from the authorities search dropdown. This patch
remedies that.

To test:
1) Ensure you have an authority with more than one word in the name,
    i.e. Electric power production
2) Go to Authorities in the staff client.
3) Change the dropdown to 'is exactly' and search for one of the words
in your record's title, i.e. 'power' in the 'main heading' search
4) Confirm your authority shows in the results, even though it is not an
exact match.
5) Search for a biblio record and go to edit the record.
6) Go to Tab 6 and click the plugin button next to one of the 6xx$a
fields to trigger the authority finder plugin
7) Repeat Step 3 and Step 4
8) Go to the OPAC and go to Authority Search
9) Repeat Step 3 and Step 4

Apply patch

10) Repeat Steps 1 to 9, confirm that this time the record does NOT show
in search results.
11) Repeat Steps 1 to 9 but this time search for the authority record's
full title, i.e. 'Electric power production'. Confirm the record does
show in the search results because the search term now matches title exactly.

Sponsored-by: Education Services Australia SCIS

Signed-off-by: David Cook <dcook@prosentient.com.au>
Comment 13 Nick Clemens 2021-04-20 18:26:26 UTC
Created attachment 119929 [details] [review]
Bug 26533: Fix 'is exactly' search for authorities

The exact search 6=3 is implemented, but it isn't used as expected when
you select 'is exactly' from the authorities search dropdown. This patch
remedies that.

To test:
1) Ensure you have an authority with more than one word in the name,
    i.e. Electric power production
2) Go to Authorities in the staff client.
3) Change the dropdown to 'is exactly' and search for one of the words
in your record's title, i.e. 'power' in the 'main heading' search
4) Confirm your authority shows in the results, even though it is not an
exact match.
5) Search for a biblio record and go to edit the record.
6) Go to Tab 6 and click the plugin button next to one of the 6xx$a
fields to trigger the authority finder plugin
7) Repeat Step 3 and Step 4
8) Go to the OPAC and go to Authority Search
9) Repeat Step 3 and Step 4

Apply patch

10) Repeat Steps 1 to 9, confirm that this time the record does NOT show
in search results.
11) Repeat Steps 1 to 9 but this time search for the authority record's
full title, i.e. 'Electric power production'. Confirm the record does
show in the search results because the search term now matches title exactly.

Sponsored-by: Education Services Australia SCIS

Signed-off-by: David Cook <dcook@prosentient.com.au>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 14 Jonathan Druart 2021-04-21 08:46:30 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 15 Fridolin Somers 2021-04-29 12:47:43 UTC
Pushed to 20.11.x for 20.11.06
Comment 16 Fridolin Somers 2021-04-29 12:57:12 UTC
This is a behavior change on a very used feature.
I suggest we do not backport on older versions.
Comment 17 Andrew Fuerste-Henry 2021-05-24 16:07:52 UTC
Pushed to 20.05.x for 20.05.12
Comment 18 Victor Grousset/tuxayo 2021-05-24 16:59:18 UTC
Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed.
Comment 19 Victor Grousset/tuxayo 2021-05-28 02:37:06 UTC
Hi, I just tried the test plan to see if bug 28475 broke this.

I'm failing step 4) even though I'm on master :o

On sample data I'm searching "AGENTES INTERESADOS" with "is exactly" in the main heading and I get two results.
When I'm searching "INTERESADOS" which "is exactly" in the main heading and I don't get a result.
(main heading: default)

Is there something missing?