Bug 22997

Summary: Searching gives no results in auth_finder.pl
Product: Koha Reporter: Fridolin SOMERS <fridolin.somers>
Component: Searching - ElasticsearchAssignee: Fridolin SOMERS <fridolin.somers>
Status: Pushed to master --- QA Contact: Martin Renvoize <martin.renvoize>
Severity: critical    
Priority: P5 - low CC: alex.arnaud, bouzid.fergani, katrin.fischer, marjorie.barry-vila, martin.renvoize, nick
Version: master   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=23004
https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=21534
Change sponsored?: --- Patch complexity: Trivial patch
Who signed the patch off: Text to go in the release notes:
Version(s) released in:
19.11.00
Bug Depends on: 21084    
Bug Blocks:    
Attachments: Bug 22997: Searching gives no results in auth_finder.pl
Bug 22997: Searching gives no results in auth_finder.pl
Bug 22997: Unit text
Bug 22997: Unit test
Bug 22997: Searching gives no results in auth_finder.pl
Bug 22997: Unit test
Bug 22997: Searching gives no results in auth_finder.pl
Bug 22997: Unit test
Bug 22997: Searching gives no results in auth_finder.pl
Bug 22997: Unit test

Description Fridolin SOMERS 2019-05-28 13:46:01 UTC
During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.
Comment 1 Fridolin SOMERS 2019-05-28 13:48:09 UTC
Looks like with a 'query_string' the wildcards are by default not detected.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

analyze_wildcard
	
By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well.
Comment 2 Fridolin SOMERS 2019-05-28 14:09:11 UTC
Created attachment 90154 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl
Comment 3 Kyle M Hall 2019-06-28 14:57:52 UTC
Created attachment 91095 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 4 Martin Renvoize 2019-07-01 15:54:01 UTC
Failing QA.. we need tests adding to Koha/SearchEngine/Elasticsearch/QueryBuilder.t for this.
Comment 5 Fridolin SOMERS 2019-08-09 11:52:32 UTC
Created attachment 92118 [details] [review]
Bug 22997: Unit text

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Comment 6 Fridolin SOMERS 2019-08-09 11:54:46 UTC
Created attachment 92119 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Comment 7 Katrin Fischer 2019-08-11 09:58:39 UTC
Counting Kyle's sign-off here, switching to signed off.
Comment 8 Katrin Fischer 2019-08-11 10:06:05 UTC
I am seeing some issues here - when I used the linking on an existing sample record I got:

Search main heading ($a only), contains: Waeschke, Hermannus,
search main headin, contains: 1850- [from old catalog]


Resulting in:

[Request] ** [http://localhost:9200]-[400] [query_shard_exception] Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])], with: {"index":"koha_kohadev_authorities","index_uuid":"S12XDtQeTMWLoBsmUsRlnw"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 100. With vars: {'body' => {'status' => 400,'error' => {'phase' => 'query','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'root_cause' => [{'type' => 'query_shard_exception','reason' => 'Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])]','index_uuid' => 'S12XDtQeTMWLoBsmUsRlnw','index' => 'koha_kohadev_authorities'}],'type' => 'search_phase_execution_exception','reason' => 'all shards failed','failed_shards' => [{'index' => 'koha_kohadev_authorities','shard' => 0,'node' => 'Secs-Ua8SyyeB_6hIXEnSA','reason' => {'type' => 'query_shard_exception','reason' => 'Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])]','index' => 'koha_kohadev_authorities','index_uuid' => 'S12XDtQeTMWLoBsmUsRlnw','caused_by' => {'caused_by' => {'type' => 'parse_exception','reason' => 'Encountered " <RANGE_GOOP> "AND "" at line 1, column 21.
Was expecting:
    "TO" ...
    '},'reason' => 'Cannot parse \'(1850-) AND ([from*) AND (old*) AND (catalog])\': Encountered " <RANGE_GOOP> "AND "" at line 1, column 21.
Was expecting:
    "TO" ...
    ','type' => 'parse_exception'}}}]}},'request' => {'serialize' => 'std','qs' => {},'path' => '/koha_kohadev_authorities/_search','method' => 'GET','ignore' => [],'mime_type' => 'application/json','body' => {'query' => {'bool' => {'must' => [{'query_string' => {'query' => '(Waeschke,) AND (Hermannus,)','default_field' => 'heading-main','analyze_wildcard' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' )}},{'query_string' => {'query' => '(1850-) AND ([from*) AND (old*) AND (catalog])','default_field' => 'heading','analyze_wildcard' => $VAR1->{'request'}{'body'}{'query'}{'bool'}{'must'}[0]{'query_string'}{'analyze_wildcard'}}}]}},'from' => 0,'size' => 20,'sort' => [{'heading__sort.phrase' => 'asc'}]}},'status_code' => 400}
Unable to perform your search. Please try again.

When I remove the second input, it works with and without the patch?
Comment 9 Nick Clemens 2019-10-01 14:48:57 UTC
Created attachment 93377 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 10 Nick Clemens 2019-10-01 14:49:01 UTC
Created attachment 93378 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 11 Nick Clemens 2019-10-01 14:50:44 UTC
The issue is searches with a '-' in the middle, like a date range
1906-1995

Square brackets '[]' actually break searching for biblios and authorities in general in ES and must be escaped, that hould be outside the scope here for the moment
Comment 12 Martin Renvoize 2019-10-09 11:50:13 UTC
Sorry Nick, this no longer applies.. I think the fix looks simple but I'm not expert enough in the area to be confident I'll have got it right.. could you rebase it please?
Comment 13 Nick Clemens 2019-10-09 12:38:39 UTC
Created attachment 93917 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 14 Nick Clemens 2019-10-09 12:38:43 UTC
Created attachment 93918 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 15 Martin Renvoize 2019-10-09 13:03:03 UTC
Created attachment 93919 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 16 Martin Renvoize 2019-10-09 13:03:07 UTC
Created attachment 93920 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 17 Martin Renvoize 2019-10-09 13:03:57 UTC
Thanks Nick, 

All looks good here (and it looks like I would have got the rebase right.. woopie)

Passing QA
Comment 18 Martin Renvoize 2019-10-09 13:32:45 UTC
Nice work!

Pushed to master for 19.11.00