Bug 25864

Summary: Case sensitivity breaks searching of some fields in ES5
Product: Koha Reporter: Nick Clemens <nick>
Component: Searching - ElasticsearchAssignee: Nick Clemens <nick>
Status: Pushed to oldstable --- QA Contact: Alex Arnaud <alex.arnaud>
Severity: major    
Priority: P5 - low CC: aleisha, alex.arnaud, andrew, bugzilla, frederic, heather_hernandez, jonathan.druart, lucas, victor
Version: master   
Hardware: All   
OS: All   
Change sponsored?: --- Patch complexity: ---
Who signed the patch off: Text to go in the release notes:
Version(s) released in:
20.11.00. 20.05.02, 19.11.08
Bug Depends on: 20589    
Bug Blocks:    
Attachments: Bug 25864: Unit test adjustment
Bug 25864 - Lower case fields for searching
Bug 25864 - Lower case fields for searching
Bug 25864: Unit test adjustment
Bug 25864 - Lower case fields for searching

Description Nick Clemens 2020-06-24 12:19:36 UTC
In Elasticsearch.pm:
1161             # Force lower case on indexed field names for case insensitive
1162             # field name searches
1163             lc($search_field->name),


But in QueryBuilder.pm
1172             push @search_fields, [
1173                 $search_field->name,

Now that we pass 'fields' for all generic searches this breaks things in ES5

To recreate:
1 - Have Koha using ES5 and Elasticsearch as search engine
2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
3 - Do a search in authorities using 'Search entire record' (abduction if using sample db)
4 - Note no results
5 - View the page source and find 'search_query'
6 - Note the uppercased fields
7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
8 - Note all fields lower-cased
Comment 1 Nick Clemens 2020-06-24 12:24:25 UTC
Created attachment 106246 [details] [review]
Bug 25864: Unit test adjustment
Comment 2 Nick Clemens 2020-06-24 12:24:28 UTC
Created attachment 106247 [details] [review]
Bug 25864 - Lower case fields for searching

To recreate:
 1 - Have Koha using ES5 and Elasticsearch as search engine
 2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
 3 - Do a search in authorities using 'Search entire record' (abduction if using sample db)
 4 - Note no results
 5 - View the page source and find 'search_query'
 6 - Note the uppercased fields
 7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
 8 - Note all fields lower-cased
 9 - Apply patch
10 - Repeat search
11 - It works!
Comment 3 Heather 2020-06-24 17:15:40 UTC
Hi!

I was unable to recreate the problem in my sandbox.  In my sandbox (which Nick made sure satisfied step 1--thank you, Nick!) I followed step 2, changing those two sysprefs from "Don't" to "Do," but then when trying step 3 with 'Search entire record' selected for the search term:
fiction
I do get results.  I also get authority record results with all four authority record search options when searching:
rowling

Am I doing something wrong to recreate the problem?

Thanks!  Cheerio!
h2

(In reply to Nick Clemens from comment #2)
> Created attachment 106247 [details] [review] [review]
> Bug 25864 - Lower case fields for searching
> 
> To recreate:
>  1 - Have Koha using ES5 and Elasticsearch as search engine
>  2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
>  3 - Do a search in authorities using 'Search entire record' (abduction if
> using sample db)
>  4 - Note no results
>  5 - View the page source and find 'search_query'
>  6 - Note the uppercased fields
>  7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
>  8 - Note all fields lower-cased
>  9 - Apply patch
> 10 - Repeat search
> 11 - It works!
Comment 4 Heather 2020-06-25 19:03:07 UTC
Replying to myself to note that my confusion is cleared up--was able to follow test plan through step 6.  Still working on it...

(In reply to Heather from comment #3)
> Hi!
> 
> I was unable to recreate the problem in my sandbox.  In my sandbox (which
> Nick made sure satisfied step 1--thank you, Nick!) I followed step 2,
> changing those two sysprefs from "Don't" to "Do," but then when trying step
> 3 with 'Search entire record' selected for the search term:
> fiction
> I do get results.  I also get authority record results with all four
> authority record search options when searching:
> rowling
> 
> Am I doing something wrong to recreate the problem?
> 
> Thanks!  Cheerio!
> h2
> 
> (In reply to Nick Clemens from comment #2)
> > Created attachment 106247 [details] [review] [review] [review]
> > Bug 25864 - Lower case fields for searching
> > 
> > To recreate:
> >  1 - Have Koha using ES5 and Elasticsearch as search engine
> >  2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
> >  3 - Do a search in authorities using 'Search entire record' (abduction if
> > using sample db)
> >  4 - Note no results
> >  5 - View the page source and find 'search_query'
> >  6 - Note the uppercased fields
> >  7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
> >  8 - Note all fields lower-cased
> >  9 - Apply patch
> > 10 - Repeat search
> > 11 - It works!
Comment 5 ByWater Sandboxes 2020-06-26 15:47:40 UTC
Created attachment 106345 [details] [review]
Bug 25864 - Lower case fields for searching

To recreate:
 1 - Have Koha using ES5 and Elasticsearch as search engine
 2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
 3 - Do a search in authorities using 'Search entire record' (abduction if using sample db)
 4 - Note no results
 5 - View the page source and find 'search_query'
 6 - Note the uppercased fields
 7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
 8 - Note all fields lower-cased
 9 - Apply patch
10 - Repeat search
11 - It works!

Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Comment 6 Heather 2020-06-26 15:49:31 UTC
Works well!  I can't wait to get this fix in our catalog!  Thank you, Nick, and everyone!!!
Comment 7 Alex Arnaud 2020-07-03 09:22:44 UTC
Created attachment 106491 [details] [review]
Bug 25864: Unit test adjustment

Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 8 Alex Arnaud 2020-07-03 09:22:48 UTC
Created attachment 106492 [details] [review]
Bug 25864 - Lower case fields for searching

To recreate:
 1 - Have Koha using ES5 and Elasticsearch as search engine
 2 - Enable DumpTemplateVarsIntranet and DumpSearchQueryTemplate
 3 - Do a search in authorities using 'Search entire record' (abduction if using sample db)
 4 - Note no results
 5 - View the page source and find 'search_query'
 6 - Note the uppercased fields
 7 - curl 'es:9200/koha_kohadev_authorities/data/417?pretty'
 8 - Note all fields lower-cased
 9 - Apply patch
10 - Repeat search
11 - It works!

Signed-off-by: Heather Hernandez <heather_hernandez@nps.gov>
Signed-off-by: Alex Arnaud <alex.arnaud@biblibre.com>
Comment 9 Jonathan Druart 2020-07-09 10:06:10 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 10 Lucas Gass 2020-07-21 15:16:57 UTC
backported to 20.05.x for 20.05.02
Comment 11 Aleisha Amohia 2020-07-22 00:04:54 UTC
backported to 19.11.x for 19.11.08
Comment 12 Victor Grousset/tuxayo 2020-07-22 22:34:00 UTC
Does look to be affecting an existing feature of 19.05.x, won't backport.

(given how things looked when trying to backport)
Comment 13 Victor Grousset/tuxayo 2020-07-26 16:33:24 UTC
*Doesn't look
Comment 14 Nick Clemens 2020-07-27 11:31:31 UTC
(In reply to Victor Grousset/tuxayo from comment #12)
> Does look to be affecting an existing feature of 19.05.x, won't backport.
> 
> (given how things looked when trying to backport)

Yes, I added dependency for 20589 - so not needed for 19.05