During cataloguing of an existing biblio, on an heading field, the use of tag editor fills authorities finder with existing value : Search main heading ($a only) Search main heading Default operator beeing 'contains'. Actually with Elasticsearch those search give no results.
Looks like with a 'query_string' the wildcards are by default not detected. See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html analyze_wildcard By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well.
Created attachment 90154 [details] [review] Bug 22997: Searching gives no results in auth_finder.pl During cataloguing of an existing biblio, on an heading field, the use of tag editor fills authorities finder with existing value : Search main heading ($a only) Search main heading Default operator beeing 'contains'. Actually with Elasticsearch those search give no results. Example with heading : 200 $a Casaubon $b Isaac $f 1559-1614 Call to Elasticsearch : "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "Casaubon*", "default_field" : "heading-main", } }, { "query_string" : { "query" : "(Isaac*) AND (1559-1614*)", "default_field" : "heading" } } ] } }, "sort" : [ { "heading__sort.phrase" : "asc" } ] } Patch adds to "query_string" : analyze_wildcard : true. Test plan : 1) Use Elasticsearch 2) Edit an existing biblio record 3) Use tag editor on a heading 4) Click search => You get correct results 5) Check also search in authorities-home.pl
Created attachment 91095 [details] [review] Bug 22997: Searching gives no results in auth_finder.pl During cataloguing of an existing biblio, on an heading field, the use of tag editor fills authorities finder with existing value : Search main heading ($a only) Search main heading Default operator beeing 'contains'. Actually with Elasticsearch those search give no results. Example with heading : 200 $a Casaubon $b Isaac $f 1559-1614 Call to Elasticsearch : "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "Casaubon*", "default_field" : "heading-main", } }, { "query_string" : { "query" : "(Isaac*) AND (1559-1614*)", "default_field" : "heading" } } ] } }, "sort" : [ { "heading__sort.phrase" : "asc" } ] } Patch adds to "query_string" : analyze_wildcard : true. Test plan : 1) Use Elasticsearch 2) Edit an existing biblio record 3) Use tag editor on a heading 4) Click search => You get correct results 5) Check also search in authorities-home.pl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Failing QA.. we need tests adding to Koha/SearchEngine/Elasticsearch/QueryBuilder.t for this.
Created attachment 92118 [details] [review] Bug 22997: Unit text Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Created attachment 92119 [details] [review] Bug 22997: Unit test Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Counting Kyle's sign-off here, switching to signed off.
I am seeing some issues here - when I used the linking on an existing sample record I got: Search main heading ($a only), contains: Waeschke, Hermannus, search main headin, contains: 1850- [from old catalog] Resulting in: [Request] ** [http://localhost:9200]-[400] [query_shard_exception] Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])], with: {"index":"koha_kohadev_authorities","index_uuid":"S12XDtQeTMWLoBsmUsRlnw"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 100. With vars: {'body' => {'status' => 400,'error' => {'phase' => 'query','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'root_cause' => [{'type' => 'query_shard_exception','reason' => 'Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])]','index_uuid' => 'S12XDtQeTMWLoBsmUsRlnw','index' => 'koha_kohadev_authorities'}],'type' => 'search_phase_execution_exception','reason' => 'all shards failed','failed_shards' => [{'index' => 'koha_kohadev_authorities','shard' => 0,'node' => 'Secs-Ua8SyyeB_6hIXEnSA','reason' => {'type' => 'query_shard_exception','reason' => 'Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])]','index' => 'koha_kohadev_authorities','index_uuid' => 'S12XDtQeTMWLoBsmUsRlnw','caused_by' => {'caused_by' => {'type' => 'parse_exception','reason' => 'Encountered " <RANGE_GOOP> "AND "" at line 1, column 21. Was expecting: "TO" ... '},'reason' => 'Cannot parse \'(1850-) AND ([from*) AND (old*) AND (catalog])\': Encountered " <RANGE_GOOP> "AND "" at line 1, column 21. Was expecting: "TO" ... ','type' => 'parse_exception'}}}]}},'request' => {'serialize' => 'std','qs' => {},'path' => '/koha_kohadev_authorities/_search','method' => 'GET','ignore' => [],'mime_type' => 'application/json','body' => {'query' => {'bool' => {'must' => [{'query_string' => {'query' => '(Waeschke,) AND (Hermannus,)','default_field' => 'heading-main','analyze_wildcard' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' )}},{'query_string' => {'query' => '(1850-) AND ([from*) AND (old*) AND (catalog])','default_field' => 'heading','analyze_wildcard' => $VAR1->{'request'}{'body'}{'query'}{'bool'}{'must'}[0]{'query_string'}{'analyze_wildcard'}}}]}},'from' => 0,'size' => 20,'sort' => [{'heading__sort.phrase' => 'asc'}]}},'status_code' => 400} Unable to perform your search. Please try again. When I remove the second input, it works with and without the patch?
Created attachment 93377 [details] [review] Bug 22997: Searching gives no results in auth_finder.pl During cataloguing of an existing biblio, on an heading field, the use of tag editor fills authorities finder with existing value : Search main heading ($a only) Search main heading Default operator beeing 'contains'. Actually with Elasticsearch those search give no results. Example with heading : 200 $a Casaubon $b Isaac $f 1559-1614 Call to Elasticsearch : "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "Casaubon*", "default_field" : "heading-main", } }, { "query_string" : { "query" : "(Isaac*) AND (1559-1614*)", "default_field" : "heading" } } ] } }, "sort" : [ { "heading__sort.phrase" : "asc" } ] } Patch adds to "query_string" : analyze_wildcard : true. Test plan : 1) Use Elasticsearch 2) Edit an existing biblio record 3) Use tag editor on a heading 4) Click search => You get correct results 5) Check also search in authorities-home.pl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93378 [details] [review] Bug 22997: Unit test Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
The issue is searches with a '-' in the middle, like a date range 1906-1995 Square brackets '[]' actually break searching for biblios and authorities in general in ES and must be escaped, that hould be outside the scope here for the moment
Sorry Nick, this no longer applies.. I think the fix looks simple but I'm not expert enough in the area to be confident I'll have got it right.. could you rebase it please?
Created attachment 93917 [details] [review] Bug 22997: Searching gives no results in auth_finder.pl During cataloguing of an existing biblio, on an heading field, the use of tag editor fills authorities finder with existing value : Search main heading ($a only) Search main heading Default operator beeing 'contains'. Actually with Elasticsearch those search give no results. Example with heading : 200 $a Casaubon $b Isaac $f 1559-1614 Call to Elasticsearch : "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "Casaubon*", "default_field" : "heading-main", } }, { "query_string" : { "query" : "(Isaac*) AND (1559-1614*)", "default_field" : "heading" } } ] } }, "sort" : [ { "heading__sort.phrase" : "asc" } ] } Patch adds to "query_string" : analyze_wildcard : true. Test plan : 1) Use Elasticsearch 2) Edit an existing biblio record 3) Use tag editor on a heading 4) Click search => You get correct results 5) Check also search in authorities-home.pl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93918 [details] [review] Bug 22997: Unit test Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Created attachment 93919 [details] [review] Bug 22997: Searching gives no results in auth_finder.pl During cataloguing of an existing biblio, on an heading field, the use of tag editor fills authorities finder with existing value : Search main heading ($a only) Search main heading Default operator beeing 'contains'. Actually with Elasticsearch those search give no results. Example with heading : 200 $a Casaubon $b Isaac $f 1559-1614 Call to Elasticsearch : "query" : { "bool" : { "must" : [ { "query_string" : { "query" : "Casaubon*", "default_field" : "heading-main", } }, { "query_string" : { "query" : "(Isaac*) AND (1559-1614*)", "default_field" : "heading" } } ] } }, "sort" : [ { "heading__sort.phrase" : "asc" } ] } Patch adds to "query_string" : analyze_wildcard : true. Test plan : 1) Use Elasticsearch 2) Edit an existing biblio record 3) Use tag editor on a heading 4) Click search => You get correct results 5) Check also search in authorities-home.pl Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com> Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Created attachment 93920 [details] [review] Bug 22997: Unit test Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t Signed-off-by: Nick Clemens <nick@bywatersolutions.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Thanks Nick, All looks good here (and it looks like I would have got the rebase right.. woopie) Passing QA
Nice work! Pushed to master for 19.11.00
Pushed to 19.05.x for 19.05.05