Bugzilla – Attachment 145606 Details for
Bug 32707
Elasticsearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields)
Home
|
New
|
Browse
|
Search
|
[?]
|
Reports
|
Help
|
New Account
|
Log In
[x]
|
Forgot Password
Login:
[x]
[patch]
[PATCH] Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields)
0001-Bug-32707-ElasticSearch-should-not-auto-truncate-even-.patch (text/plain), 6.11 KB, created by
Janusz Kaczmarek
on 2023-01-24 10:56:49 UTC
(
hide
)
Description:
[PATCH] Bug 32707: ElasticSearch should not auto truncate (even if QueryAutoTruncate = 1) for identifiers (and some other fields)
Filename:
MIME Type:
Creator:
Janusz Kaczmarek
Created:
2023-01-24 10:56:49 UTC
Size:
6.11 KB
patch
obsolete
>From 63d68a9a2e2ab837f8cbcff0ecc9bb0c23d5c98f Mon Sep 17 00:00:00 2001 >From: Janusz Kaczmarek <januszop@gmail.com> >Date: Tue, 24 Jan 2023 10:45:39 +0100 >Subject: [PATCH] Bug 32707: ElasticSearch should not auto truncate (even if > QueryAutoTruncate = 1) for identifiers (and some other fields) >MIME-Version: 1.0 >Content-Type: text/plain; charset=UTF-8 >Content-Transfer-Encoding: 8bit > >Koha with Zebra prevented auto truncation in some circumstances -- see >the first return for ccl inside C4::Search:: buildQuery, before applying >auto truncation, and setting $auto_truncation = 0 for some search >fields. > >Koha with ElasticSearch applies auto truncation for all search fields, >not paying attention to these special cases when it should not be done. >This leads to various problems as described in bug 26508, 26608, etc. > >The solution would be to prevent auto truncation for certain search >fields, above all รข the identifiers. In addition, under no >circumstances should a search field other than of text type be truncated >(an attempt to truncate would generate an exception from ElasticSearch, >e.g. number_format_exception for integer). > >Test plan >========= >Scenario A (authority) >---------------------- >1. Enable Elasticsearch engine. >2. Have like 100+ bibliographic records and properly linked authority > records. >3. Reindex ElasticSearch if needed. >4. Enable QueryAutoTruncate systempreference. >5. Find the authority record #1 and note the number of linked biblio > records. >6. Show the detail of authority #1 and compare the number of linked > biblio records. >7. If in the database there are authority records with ids =~ /^1/ (i.e. > 10, 11, 12, ..., 100, 101, ...) linked to the biblio records you get > two different numbers of linked records. >8. Also, as lists of linked biblio records (via link: Used in N > record(s) from results view and detail view) you will get two > different sets of biblio records. In particular, on the list generated > from detail view (authorities/detail.pl?authid=1) you will get biblio > records that are in fact not linked to the auth #1 (the list is > generated with &q=an:1). >9. Apply this patch. >10. Counts and list of linked biblios should be ok. > >Scenario B (analytics) >---------------------- >1. Enable Elasticsearch engine. >2. Have three monographic bibliographic records with 001 = 1, 10, 100 > (i.e. =~ /^1/). >3. Have an analytical record with 773 $w = 1. >4. Reindex ElasticSearch if needed. >5. Enable QueryAutoTruncate and UseControlNumber systempreference. >6. Find the analytical record and click on the link generated from 773, > i.e. (In: ... --> catalogue/search.pl?q=Control-number:1). >7. You should see a list of three records (001 = 1, 10, 100) instead of > one (001 = 1). >8. Apply this patch. >9. Repeat step 6. You should arrive at the right host record. >--- > .../Elasticsearch/QueryBuilder.pm | 52 ++++++++++++++++++- > 1 file changed, 51 insertions(+), 1 deletion(-) > >diff --git a/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm b/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm >index 53042b4303..fcb3953365 100644 >--- a/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm >+++ b/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm >@@ -293,7 +293,7 @@ sub build_query_compat { > while ( my ( $oand, $otor, $index ) = $ea->() ) { > next if ( !defined($oand) || $oand eq '' ); > $oand = $self->clean_search_term($oand); >- $oand = $self->_truncate_terms($oand) if ($truncate); >+ $oand = $self->_truncate_terms($oand) if $truncate && $self->_is_safe_to_auto_truncate($index->{field}, $oand); > push @search_params, { > operand => $oand, # the search terms > operator => defined($otor) ? uc $otor : undef, # AND and so on >@@ -1305,4 +1305,54 @@ sub _search_fields { > } > } > >+=pod >+ >+=head2 _is_safe_to_auto_truncate >+ >+_is_safe_to_auto_truncate($index_field, $oand); >+ >+Checks if it is safe to auto truncate a search term within a given search field. >+ >+The search term should not be auto truncated when searching for identifiers, e.g. >+koha-auth-number, record-control-number, local-number etc. Also, non-text fields >+must not be auto truncated (doing so would generate ES exception). >+ >+=cut >+ >+sub _is_safe_to_auto_truncate { >+ my ($self, $index_field, $oand) = @_; >+ # Do not auto truncate fields that should not be auto truncated, >+ # primarily various types of identifiers, above all record identifiers. >+ # Alternatively this could be moved to the ElasticSearch search fields >+ # configuration page. >+ my @do_not_truncate = qw / barcode biblioitemnumber control-number control-number-identifier date-of-acquisition date-of-publication date-time-last-modified host-item-number identifier-standard isbn issn itemnumber itype koha-auth-number lc-card-number local-number number-local-acquisition other-control-number record-control-number /; >+ # In addition, under no circumstances should non-text fields >+ # be auto truncated. >+ my $cache = Koha::Caches->get_instance(); >+ my $cache_key = 'elasticsearch_search_non_text_fields'; >+ my $non_text_search_fields = $cache->get_from_cache($cache_key, { unsafe => 1 }); >+ if (!$non_text_search_fields) { >+ my $schema = Koha::Database->new()->schema(); >+ my $sfs = $schema->resultset('SearchField')->search({ '-not' => { '-or' => [ {type => ''}, {type => 'string'} ] } }); >+ while (my $sf = $sfs->next) { >+ push @$non_text_search_fields, $sf->name; >+ } >+ $cache->set_in_cache($cache_key, $non_text_search_fields); >+ } >+ push @do_not_truncate, @$non_text_search_fields if $non_text_search_fields; >+ >+ # search fields can be given as a explicit index name (e.g. from advanced >+ # search): >+ if ($index_field) { >+ return 0 if grep { $index_field eq $_ } @do_not_truncate; >+ # OR can be given implicite, as prefix in the operand (e.g. in links generated >+ # by Koha like catalogue/search.pl?q=an:<authid>): >+ } elsif ($oand =~ /\b($field_name_pattern):/) { # check field name prefixing operand >+ my $field_name = $1; >+ return 0 if grep { $field_name eq $_ } @do_not_truncate; >+ } >+ # It safe to auto truncate: >+ return 1; >+} >+ > 1; >-- >2.30.2 >
You cannot view the attachment while viewing its details because your browser does not support IFRAMEs.
View the attachment on a separate page
.
View Attachment As Diff
View Attachment As Raw
Actions:
View
|
Diff
|
Splinter Review
Attachments on
bug 32707
:
145606
|
158007
|
158008
|
158009
|
161035
|
163748
|
163749
|
163750
|
163751
|
163759
|
163822
|
163823
|
163824
|
163880
|
163881
|
163882
|
163892
|
163893
|
163894
|
164140
|
164141
|
164142