Bug 31023 - Cannot create new GENRE/FORM authorities when QueryRegexEscapeOptions set to 'Unescape escaped'
Summary: Cannot create new GENRE/FORM authorities when QueryRegexEscapeOptions set t...
Status: RESOLVED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low normal (vote)
Assignee: Nick Clemens
QA Contact:
URL:
Keywords:
Depends on:
Blocks:
 
Reported: 2022-06-23 10:08 UTC by Nick Clemens
Modified: 2022-11-20 00:22 UTC (History)
4 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Small patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
22.11.00, 22.05.07, 21.11.14


Attachments
Bug 31023: Unit tests (2.50 KB, patch)
2022-07-12 13:48 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 31023: Quote authtypecode in FindDuplicateAuthorities (1.83 KB, patch)
2022-07-12 13:48 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 31023: Unit tests (2.55 KB, patch)
2022-07-12 20:27 UTC, David Nind
Details | Diff | Splinter Review
Bug 31023: Quote authtypecode in FindDuplicateAuthorities (1.88 KB, patch)
2022-07-12 20:27 UTC, David Nind
Details | Diff | Splinter Review
Bug 31023: Unit tests (2.61 KB, patch)
2022-08-27 13:06 UTC, Katrin Fischer
Details | Diff | Splinter Review
Bug 31023: Quote authtypecode in FindDuplicateAuthorities (1.93 KB, patch)
2022-08-27 13:06 UTC, Katrin Fischer
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens 2022-06-23 10:08:29 UTC
Bug 24286 and bug 24264 added some escaping for the forward slash in this authtype, however, when we unescape escaped characters for regex, it appears we break the escaping in FindDuplicateAuthority

To test:
0 - Have Koha using Elasticsearch
1 - Set  QueryRegexEscapeOptions to 'unescape escaped'
2 - Attempt to add a new 'GENRE/FORM' authority record
3 - On save you get a 500 error
Unable to understand your search query, please rephrase and try again.
 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 102

[2022/06/23 10:07:02] [WARN] [Request] ** [http://es:9200]-[400] [query_shard_exception] Failed to parse query [(authtype:GENRE/FORM* AND heading:"ddddd")], with: {"index":"koha_kohadev_authorities","index_uuid":"I00T6jgPTKynx5kKc8heNQ"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 95. With vars: {'request' => {'body' => {'query' => {'query_string' => {'query' => '(authtype:GENRE/FORM* AND heading:"ddddd")','analyze_wildcard' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'default_operator' => 'AND','fuzziness' => 'auto','type' => 'cross_fields','lenient' => $VAR1->{'request'}{'body'}{'query'}{'query_string'}{'analyze_wildcard'},'fields' => ['lc-card-number','local-number','record-source','personal-name','meeting-name','personal-name-see-also-from','see-from','authtype','meeting-name-see-also-from','match-heading-see-from','heading-use-main-or-added-entry','subject-heading-thesaurus','match','heading-use-series-added-entry','kind-of-record','see-also-from','heading','corporate-name-see-also-from','descriptive-cataloging-rules','meeting-name-see-from','corporate-name-see-from','heading-use-subject-added-entry','personal-name-see-from','personal-name-heading','meeting-name-heading','heading-main']}},'size' => 1,'aggregations' => {'su-geo' => {'terms' => {'size' => '20','field' => 'su-geo__facet'}},'location' => {'terms' => {'size' => '20','field' => 'location__facet'}},'author' => {'terms' => {'field' => 'author__facet','size' => '20'}},'ln' => {'terms' => {'field' => 'ln__facet','size' => '20'}},'holdingbranch' => {'terms' => {'size' => '20','field' => 'holdingbranch__facet'}},'subject' => {'terms' => {'field' => 'subject__facet','size' => '20'}},'itype' => {'terms' => {'field' => 'itype__facet','size' => '20'}},'title-series' => {'terms' => {'field' => 'title-series__facet','size' => '20'}},'ccode' => {'terms' => {'field' => 'ccode__facet','size' => '20'}}},'from' => 0},'mime_type' => 'application/json','serialize' => 'std','method' => 'GET','ignore' => [],'path' => '/koha_kohadev_authorities/_search','qs' => {'track_total_hits' => 'true'}},'status_code' => 400,'body' => {'error' => {'reason' => 'all shards failed','phase' => 'query','type' => 'search_phase_execution_exception','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'failed_shards' => [{'node' => '8aathbKCRVmwFPakemaNEA','index' => 'koha_kohadev_authorities','reason' => {'reason' => 'Failed to parse query [(authtype:GENRE/FORM* AND heading:"ddddd")]','index' => 'koha_kohadev_authorities','type' => 'query_shard_exception','caused_by' => {'type' => 'parse_exception','caused_by' => {'type' => 'token_mgr_error','reason' => 'Lexical error at line 1, column 43.  Encountered: <EOF> after : "/FORM* AND heading:\\"ddddd\\")"'},'reason' => 'Cannot parse \'(authtype:GENRE/FORM* AND heading:"ddddd")\': Lexical error at line 1, column 43.  Encountered: <EOF> after : "/FORM* AND heading:\\"ddddd\\")"'},'index_uuid' => 'I00T6jgPTKynx5kKc8heNQ'},'shard' => 0}],'root_cause' => [{'index' => 'koha_kohadev_authorities','reason' => 'Failed to parse query [(authtype:GENRE/FORM* AND heading:"ddddd")]','index_uuid' => 'I00T6jgPTKynx5kKc8heNQ','type' => 'query_shard_exception'}]},'status' => 400}}
Comment 1 Nick Clemens 2022-07-12 13:48:27 UTC
Created attachment 137631 [details] [review]
Bug 31023: Unit tests
Comment 2 Nick Clemens 2022-07-12 13:48:31 UTC
Created attachment 137632 [details] [review]
Bug 31023: Quote authtypecode in FindDuplicateAuthorities

To test:
0 - Have Koha using Elasticsearch
1 - Set  QueryRegexEscapeOptions to 'unescape escaped'
2 - Attempt to add a new 'GENRE/FORM' authority record
3 - On save you get a 500 error
Unable to understand your search query, please rephrase and try again.
 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 102
4 - Apply patch, restart all
5 - Attempt to add a new 'GENRE/FORM' authority record
6 - Success!
Comment 3 David Nind 2022-07-12 20:27:41 UTC
Created attachment 137662 [details] [review]
Bug 31023: Unit tests

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 David Nind 2022-07-12 20:27:48 UTC
Created attachment 137663 [details] [review]
Bug 31023: Quote authtypecode in FindDuplicateAuthorities

To test:
0 - Have Koha using Elasticsearch
1 - Set  QueryRegexEscapeOptions to 'unescape escaped'
2 - Attempt to add a new 'GENRE/FORM' authority record
3 - On save you get a 500 error
Unable to understand your search query, please rephrase and try again.
 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 102
4 - Apply patch, restart all
5 - Attempt to add a new 'GENRE/FORM' authority record
6 - Success!

Signed-off-by: David Nind <david@davidnind.com>
Comment 5 Katrin Fischer 2022-07-29 08:20:50 UTC
Nick, please don't forget to assign yourself!
Comment 6 Katrin Fischer 2022-08-27 13:06:29 UTC
Created attachment 139852 [details] [review]
Bug 31023: Unit tests

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 7 Katrin Fischer 2022-08-27 13:06:33 UTC
Created attachment 139853 [details] [review]
Bug 31023: Quote authtypecode in FindDuplicateAuthorities

To test:
0 - Have Koha using Elasticsearch
1 - Set  QueryRegexEscapeOptions to 'unescape escaped'
2 - Attempt to add a new 'GENRE/FORM' authority record
3 - On save you get a 500 error
Unable to understand your search query, please rephrase and try again.
 at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Search.pm line 102
4 - Apply patch, restart all
5 - Attempt to add a new 'GENRE/FORM' authority record
6 - Success!

Signed-off-by: David Nind <david@davidnind.com>

Signed-off-by: Katrin Fischer <katrin.fischer.83@web.de>
Comment 8 Tomás Cohen Arazi 2022-08-31 11:39:31 UTC
Hi, Nick. Do you think you could use Koha::SearchEngine::Zebra::QueryBuilder::clean_search_term on the authtypecode?
Comment 9 Nick Clemens 2022-09-02 06:47:26 UTC
(In reply to Tomás Cohen Arazi from comment #8)
> Hi, Nick. Do you think you could use
> Koha::SearchEngine::Zebra::QueryBuilder::clean_search_term on the
> authtypecode?

I think that might be overkill on this one - everythign already works in Zebra - in ES, it only doesn't work under a special case where we are unescaping values as specified by the pref.

Ideally, I think we would just remove the slash from the type, GENRE_FORM, or something.
Comment 10 Tomás Cohen Arazi 2022-09-07 20:58:06 UTC
Pushed to master for 22.11.

Nice work everyone, thanks!
Comment 11 Lucas Gass 2022-10-31 20:09:41 UTC
Backported to 22.05.x for upcoming 22.05.07 release
Comment 12 Arthur Suzuki 2022-11-14 13:35:06 UTC
applied to 21.11.x for 21.11.14
Comment 13 Victor Grousset/tuxayo 2022-11-20 00:22:31 UTC
Not backported to oldoldstable (21.05.x). Feel free to ask if it's needed.

Nothing to document it seems, marking resolved.