Bug 22997 - Searching gives no results in auth_finder.pl
Summary: Searching gives no results in auth_finder.pl
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: master
Hardware: All All
: P5 - low critical with 10 votes (vote)
Assignee: Fridolin Somers
QA Contact: Martin Renvoize
URL:
Keywords:
Depends on: 21084
Blocks:
  Show dependency treegraph
 
Reported: 2019-05-28 13:46 UTC by Fridolin Somers
Modified: 2021-06-14 21:29 UTC (History)
6 users (show)

See Also:
Change sponsored?: ---
Patch complexity: Trivial patch
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
19.11.00,19.05.05


Attachments
Bug 22997: Searching gives no results in auth_finder.pl (2.24 KB, patch)
2019-05-28 14:09 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 22997: Searching gives no results in auth_finder.pl (2.31 KB, patch)
2019-06-28 14:57 UTC, Kyle M Hall
Details | Diff | Splinter Review
Bug 22997: Unit text (1.24 KB, patch)
2019-08-09 11:52 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 22997: Unit test (1.24 KB, patch)
2019-08-09 11:54 UTC, Fridolin Somers
Details | Diff | Splinter Review
Bug 22997: Searching gives no results in auth_finder.pl (2.35 KB, patch)
2019-10-01 14:48 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22997: Unit test (1.30 KB, patch)
2019-10-01 14:49 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22997: Searching gives no results in auth_finder.pl (2.56 KB, patch)
2019-10-09 12:38 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22997: Unit test (1.29 KB, patch)
2019-10-09 12:38 UTC, Nick Clemens
Details | Diff | Splinter Review
Bug 22997: Searching gives no results in auth_finder.pl (2.63 KB, patch)
2019-10-09 13:03 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 22997: Unit test (1.35 KB, patch)
2019-10-09 13:03 UTC, Martin Renvoize
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Fridolin Somers 2019-05-28 13:46:01 UTC
During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.
Comment 1 Fridolin Somers 2019-05-28 13:48:09 UTC
Looks like with a 'query_string' the wildcards are by default not detected.

See https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html

analyze_wildcard
	
By default, wildcards terms in a query string are not analyzed. By setting this value to true, a best effort will be made to analyze those as well.
Comment 2 Fridolin Somers 2019-05-28 14:09:11 UTC
Created attachment 90154 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl
Comment 3 Kyle M Hall 2019-06-28 14:57:52 UTC
Created attachment 91095 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Comment 4 Martin Renvoize 2019-07-01 15:54:01 UTC
Failing QA.. we need tests adding to Koha/SearchEngine/Elasticsearch/QueryBuilder.t for this.
Comment 5 Fridolin Somers 2019-08-09 11:52:32 UTC
Created attachment 92118 [details] [review]
Bug 22997: Unit text

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Comment 6 Fridolin Somers 2019-08-09 11:54:46 UTC
Created attachment 92119 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t
Comment 7 Katrin Fischer 2019-08-11 09:58:39 UTC
Counting Kyle's sign-off here, switching to signed off.
Comment 8 Katrin Fischer 2019-08-11 10:06:05 UTC
I am seeing some issues here - when I used the linking on an existing sample record I got:

Search main heading ($a only), contains: Waeschke, Hermannus,
search main headin, contains: 1850- [from old catalog]


Resulting in:

[Request] ** [http://localhost:9200]-[400] [query_shard_exception] Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])], with: {"index":"koha_kohadev_authorities","index_uuid":"S12XDtQeTMWLoBsmUsRlnw"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /home/vagrant/kohaclone/Koha/SearchEngine/Elasticsearch/Search.pm line 100. With vars: {'body' => {'status' => 400,'error' => {'phase' => 'query','grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'root_cause' => [{'type' => 'query_shard_exception','reason' => 'Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])]','index_uuid' => 'S12XDtQeTMWLoBsmUsRlnw','index' => 'koha_kohadev_authorities'}],'type' => 'search_phase_execution_exception','reason' => 'all shards failed','failed_shards' => [{'index' => 'koha_kohadev_authorities','shard' => 0,'node' => 'Secs-Ua8SyyeB_6hIXEnSA','reason' => {'type' => 'query_shard_exception','reason' => 'Failed to parse query [(1850-) AND ([from*) AND (old*) AND (catalog])]','index' => 'koha_kohadev_authorities','index_uuid' => 'S12XDtQeTMWLoBsmUsRlnw','caused_by' => {'caused_by' => {'type' => 'parse_exception','reason' => 'Encountered " <RANGE_GOOP> "AND "" at line 1, column 21.
Was expecting:
    "TO" ...
    '},'reason' => 'Cannot parse \'(1850-) AND ([from*) AND (old*) AND (catalog])\': Encountered " <RANGE_GOOP> "AND "" at line 1, column 21.
Was expecting:
    "TO" ...
    ','type' => 'parse_exception'}}}]}},'request' => {'serialize' => 'std','qs' => {},'path' => '/koha_kohadev_authorities/_search','method' => 'GET','ignore' => [],'mime_type' => 'application/json','body' => {'query' => {'bool' => {'must' => [{'query_string' => {'query' => '(Waeschke,) AND (Hermannus,)','default_field' => 'heading-main','analyze_wildcard' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' )}},{'query_string' => {'query' => '(1850-) AND ([from*) AND (old*) AND (catalog])','default_field' => 'heading','analyze_wildcard' => $VAR1->{'request'}{'body'}{'query'}{'bool'}{'must'}[0]{'query_string'}{'analyze_wildcard'}}}]}},'from' => 0,'size' => 20,'sort' => [{'heading__sort.phrase' => 'asc'}]}},'status_code' => 400}
Unable to perform your search. Please try again.

When I remove the second input, it works with and without the patch?
Comment 9 Nick Clemens 2019-10-01 14:48:57 UTC
Created attachment 93377 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 10 Nick Clemens 2019-10-01 14:49:01 UTC
Created attachment 93378 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 11 Nick Clemens 2019-10-01 14:50:44 UTC
The issue is searches with a '-' in the middle, like a date range
1906-1995

Square brackets '[]' actually break searching for biblios and authorities in general in ES and must be escaped, that hould be outside the scope here for the moment
Comment 12 Martin Renvoize 2019-10-09 11:50:13 UTC
Sorry Nick, this no longer applies.. I think the fix looks simple but I'm not expert enough in the area to be confident I'll have got it right.. could you rebase it please?
Comment 13 Nick Clemens 2019-10-09 12:38:39 UTC
Created attachment 93917 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 14 Nick Clemens 2019-10-09 12:38:43 UTC
Created attachment 93918 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Comment 15 Martin Renvoize 2019-10-09 13:03:03 UTC
Created attachment 93919 [details] [review]
Bug 22997: Searching gives no results in auth_finder.pl

During cataloguing of an existing biblio, on an heading field,
the use of tag editor fills authorities finder with existing value :
  Search main heading ($a only)
  Search main heading
Default operator beeing 'contains'.

Actually with Elasticsearch those search give no results.

Example with heading :
200
  $a Casaubon
  $b Isaac
  $f 1559-1614

Call to Elasticsearch :
    "query" : {
        "bool" : {
           "must" : [
              {
             "query_string" : {
                  "query" : "Casaubon*",
                  "default_field" : "heading-main",
                   }
              },
              {
             "query_string" : {
                  "query" : "(Isaac*) AND (1559-1614*)",
                  "default_field" : "heading"
                   }
              }
            ]
         }
      },
    "sort" : [
       {
      "heading__sort.phrase" : "asc"
       }
     ]
  }

Patch adds to "query_string" :
  analyze_wildcard : true.

Test plan :
1) Use Elasticsearch
2) Edit an existing biblio record
3) Use tag editor on a heading
4) Click search => You get correct results
5) Check also search in authorities-home.pl

Signed-off-by: Kyle M Hall <kyle@bywatersolutions.com>
Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 16 Martin Renvoize 2019-10-09 13:03:07 UTC
Created attachment 93920 [details] [review]
Bug 22997: Unit test

Run prove t/db_dependent/Koha/SearchEngine/Elasticsearch/QueryBuilder.t

Signed-off-by: Nick Clemens <nick@bywatersolutions.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 17 Martin Renvoize 2019-10-09 13:03:57 UTC
Thanks Nick, 

All looks good here (and it looks like I would have got the rebase right.. woopie)

Passing QA
Comment 18 Martin Renvoize 2019-10-09 13:32:45 UTC
Nice work!

Pushed to master for 19.11.00
Comment 19 Fridolin Somers 2019-11-08 14:38:32 UTC
Pushed to 19.05.x for 19.05.05