Bug 39079 - Matchpoints with multiple fields require all fields to match under Elasticsearch
Summary: Matchpoints with multiple fields require all fields to match under Elasticsearch
Status: Needs Signoff
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Nick Clemens (kidclamp)
QA Contact: Testopia
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2025-02-10 17:04 UTC by Nick Clemens (kidclamp)
Modified: 2025-02-16 23:47 UTC (History)
2 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:


Attachments
Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat (5.51 KB, patch)
2025-02-10 17:53 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2025-02-10 17:04:41 UTC
In zebra when matching an incoming file to existing records, we use 'OR' to combine the fields

In elasticsearch we are combining multiple fields with 'AND' as a 'must' search

This means an incoming record with two control numbers will not match an existing record with only one


The generated queries are like:
2025-02-10T14:59:07 zebrasrv(46) [request] Search authorities OK 0 1 1+0 RPN @attrset Bib-1 @or @attr 1=Local-Number @attr 4=109 @attr 7=1 0 @or @or @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=other-control-number (DE-588)4179998-7 @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=other-control-number (DE-627)104628669 @attr 6=3 @attr 5=100 @attr 4=1 @attr 1=other-control-number (DE-576)209995106


versus
                       'bool' => {
                                   'must' => [
                                               {
                                                 'term' => {
                                                             'other-control-number.ci_raw' => '(DE-588)4179998-7'
                                                           }
                                               },
                                               {
                                                 'term' => {
                                                             'other-control-number.ci_raw' => '(DE-627)104628669'
                                                           }
                                               },
                                               {
                                                 'term' => {
                                                             'other-control-number.ci_raw' => '(DE-576)209995106'
                                                           }
                                               }
                                             ]
                                 }
                     }



This is due to this bit of code:
ElasticSearch/QueryBuilder:
 469     # Merge the query parts appropriately
 470     # 'should' behaves like 'or'
 471     # 'must' behaves like 'and'
 472     # Zebra behaviour seem to match must so using that here
 473     my $elastic_query = {};
 474     $elastic_query->{bool}->{must} = \@query_parts;


We need to respect the and_or passed into build_auth_query_compat, but currently:
 514 =item and_or
 515 
 516 Totally ignored. It is never used in L<C4::AuthoritiesMarc::SearchAuthorities>.
Comment 1 Nick Clemens (kidclamp) 2025-02-10 17:53:36 UTC
Created attachment 177706 [details] [review]
Bug 39079: Follow 'and_or' variable when passed to build_auth_query_compat

When building queries for matchign Zebra uses OR to combine searches - Elasticsearch
has been effectively using 'AND' - this patch checks whether and_or has been passed
and follows the variable when present.

To test:
1 - Have an import file of auths with multiple 035
2 - Setup an authority matching rule
    Search index: other-control-number
    Score: 1000
    Tag: 035
    Subfields: a
3 - Make sure Koha is using Elasticsearch
4 - Stage file without matching, import file
5 - Stage file again, using your matching rule
6 - Note all records match
7 - Setup Cataloging->Marc modification template
8 - Add an action to delete first 035 field
9 - Run it against the records you just imported
    You can use a report:
    SELECT authid FROM auth_header WHERE authid > #### (find the lowest new record number in batch you imported)
10 - Change matching rule in batch to none, then back to your rule
11 - No matches!
12 - Apply patch, restart all
13 - Set matchign rule to none, then back to your rule
14 - Matches!