Bugzilla – Attachment 76977 Details for
Bug 19604
Elasticsearch Fixes for build_authorities_query for auth searching
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
Bug 19604: Elasticsearch Fixes for build_authorities_query for auth searching
Bug-19604-Elasticsearch-Fixes-for-buildauthorities.patch (text/plain), 6.14 KB, created by
Nick Clemens (kidclamp)
on 2018-07-16 11:33:53 UTC
(
hide
)
Description:
Bug 19604: Elasticsearch Fixes for build_authorities_query for auth searching
Filename:
MIME Type:
Creator:
Nick Clemens (kidclamp)
Created:
2018-07-16 11:33:53 UTC
Size:
6.14 KB
patch
obsolete
>From fd2617c5870685d1f5fdab8758356089b940d575 Mon Sep 17 00:00:00 2001 >From: Nick Clemens <nick@bywatersolutions.com> >Date: Thu, 9 Nov 2017 16:23:57 +0000 >Subject: [PATCH] Bug 19604: Elasticsearch Fixes for build_authorities_query > for auth searching > >To test: >1 - Do some authority searches in Zebra >2 - Switch to ES and repeat, results will vary and some may fail >3 - Apply patch and dependencies >4 - Reindex ES >5 - Repeat searches, they should suceed and results should be similar to >Zebra >6 - Slight differences are okay, but results should (mostly) meet >expectations > >A few notes: >We add a 'normalizer' to ensure we get a single token from the heading >indexes, this makes 'starts with' work as expcted >We switch to 'AND' for fields searched from cataloging editor - this >matches Zebra results >We force the '__sort' fields for sorting - if sorting looks wrong try >reducing the heading field to a single subfield - this will need to be >addressed on a future bug (multiple subfields create an array, ES sorts > those randomly) > >Signed-off-by: Nicolas Legrand <nicolas.legrand@bulac.fr> >--- > Koha/SearchEngine/Elasticsearch/QueryBuilder.pm | 55 ++++++++++------------ > admin/searchengine/elasticsearch/field_config.yaml | 3 ++ > admin/searchengine/elasticsearch/index_config.yaml | 3 ++ > 3 files changed, 31 insertions(+), 30 deletions(-) > >diff --git a/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm b/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm >index 8a0ead1..5d7136c 100644 >--- a/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm >+++ b/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm >@@ -290,40 +290,46 @@ sub build_authorities_query { > > # Start by making the query parts > my @query_parts; >- my @filter_parts; >+ > foreach my $s ( @{ $search->{searches} } ) { > my ( $wh, $op, $val ) = @{$s}{qw(where operator value)}; > $wh = '_all' if $wh eq ''; > if ( $op eq 'is' || $op eq '=' ) { > >- # look for something that matches completely >+ # look for something that matches a term completely > # note, '=' is about numerical vals. May need special handling. >- # _allphrase is a special field that only groups the exact >- # matches. Also, we lowercase our search because the ES >+ # Also, we lowercase our search because the ES > # index lowercases its values, and term searches don't get the > # search analyzer applied to them. >- push @filter_parts, { term => { "$wh.phrase" => lc $val } }; >+ push @query_parts, { term => {"$wh.phrase" => lc $val} }; > } > elsif ( $op eq 'exact' ) { >- > # left and right truncation, otherwise an exact phrase >- push @query_parts, { match_phrase => { $wh => $val } }; >+ push @query_parts, { match_phrase => {"$wh.phrase" => lc $val} }; > } > elsif ( $op eq 'start' ) { >- >- # startswith search >- push @query_parts, { wildcard => { "$wh.phrase" => lc "$val*" } }; >+ # startswith search, uses lowercase untokenized version of heading >+ push @query_parts, { prefix => {"$wh.lc_raw" => lc $val} }; > } > else { > # regular wordlist stuff >- push @query_parts, { match => { $wh => $val } }; >+# push @query_parts, { match => {$wh => { query => $val, operator => 'and' }} }; >+ my @values = split(' ',$val); >+ foreach $val (@values) { >+ push @query_parts, { wildcard => { "$wh.phrase" => "*" . lc $val . "*" } }; >+ } > } > } > >- # Merge the query and filter parts appropriately >- # 'should' behaves like 'or', if we want 'and', use 'must' >- my $query_part = { bool => { should => \@query_parts } }; >- my $filter_part = { bool => { should => \@filter_parts } }; >+ # Merge the query parts appropriately >+ # 'should' behaves like 'or' >+ # 'must' behaves like 'and' >+ # Zebra results seem to match must so using that here >+ my $query = { query=> >+ { bool => >+ { must => \@query_parts } >+ } >+ }; > > # We need to add '.phrase' to all the sort headings otherwise it'll sort > # based on the tokenised form. >@@ -336,20 +342,9 @@ sub build_authorities_query { > $search->{sort} = \%s; > } > >- # extract the sort stuff >- my %sort; >- %sort = ( sort => [ $search->{sort} ] ) if exists $search->{sort}; >- my $query; >- if (@filter_parts) { >- $query = >- { query => >- { filtered => { filter => $filter_part, query => $query_part } } >- }; >- } >- else { >- $query = { query => $query_part }; >- } >- $query = { %$query, %sort }; >+ # add the sort stuff >+ $query->{sort} = [ $search->{sort} ] if exists $search->{sort}; >+ > return $query; > } > >@@ -446,7 +441,7 @@ sub build_authorities_query_compat { > > my %sort; > my $sort_field = >- ( $orderby =~ /^Heading/ ) ? 'Heading' >+ ( $orderby =~ /^Heading/ ) ? 'Heading__sort' > : ( $orderby =~ /^Auth/ ) ? 'Local-Number' > : undef; > if ($sort_field) { >diff --git a/admin/searchengine/elasticsearch/field_config.yaml b/admin/searchengine/elasticsearch/field_config.yaml >index 535a43c..dba47f5 100644 >--- a/admin/searchengine/elasticsearch/field_config.yaml >+++ b/admin/searchengine/elasticsearch/field_config.yaml >@@ -39,6 +39,9 @@ search: > search_analyzer: analyser_phrase > raw: > type: keyword >+ lc_raw: >+ type: keyword >+ normalizer: my_normalizer > copy_to: _all > # Facets > facet: >diff --git a/admin/searchengine/elasticsearch/index_config.yaml b/admin/searchengine/elasticsearch/index_config.yaml >index bdfcdf5..dce27bc 100644 >--- a/admin/searchengine/elasticsearch/index_config.yaml >+++ b/admin/searchengine/elasticsearch/index_config.yaml >@@ -25,6 +25,9 @@ index: > type: custom > filter: > - icu_folding >+ my_normalizer: >+ type: custom >+ char_filter: icu_normalizer > char_filter: > # The punctuation filter is used to remove any punctuation chars in fields that don't use icu_tokenizer. > punctuation: >-- >2.1.4
<b>You cannot view the attachment while viewing its details because your browser does not support IFRAMEs. <a href="attachment.cgi?id=76977">View the attachment on a separate page</a>.</b>
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 19604
:
69072
|
69198
|
69199
|
72807
|
72808
|
76977
|
76978
|
77088
|
77089
|
77328