Bug 24567 - Elasticsearch: CCL syntax does not allow for multiple indexes to be searched at once
Summary: Elasticsearch: CCL syntax does not allow for multiple indexes to be searched ...
Status: CLOSED FIXED
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Nick Clemens (kidclamp)
QA Contact:
URL:
Keywords:
: 24372 25263 (view as bug list)
Depends on:
Blocks:
 
Reported: 2020-02-03 14:47 UTC by Nick Clemens (kidclamp)
Modified: 2022-06-06 20:24 UTC (History)
9 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:
20.11.00,20.05.10
Circulation function:


Attachments
Bug 24567: Don't strip spaces along with dangling colons (3.04 KB, patch)
2020-10-27 11:31 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 24567: Don't strip spaces along with dangling colons (3.17 KB, patch)
2020-10-27 12:30 UTC, Séverine Queune
Details | Diff | Splinter Review
Bug 24567: (follow-up) Use dollar sign to refer to captures (906 bytes, patch)
2020-11-02 13:44 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 24567: Don't strip spaces along with dangling colons (3.25 KB, patch)
2020-11-03 09:54 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 24567: (follow-up) Use dollar sign to refer to captures (1.03 KB, patch)
2020-11-03 09:54 UTC, Joonas Kylmälä
Details | Diff | Splinter Review
Bug 24567: (QA follow-up) Remove warning in regex (1.36 KB, patch)
2020-11-03 10:35 UTC, Joonas Kylmälä
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Nick Clemens (kidclamp) 2020-02-03 14:47:00 UTC
A search like:
ti:glory and au:clinton

should return results with glory in the title and clinton in the author, this works in Zebra but does not work in ES currently
Comment 1 Nick Clemens (kidclamp) 2020-02-26 02:41:04 UTC
AND must be capitalized, but otherwise not sure what I found here, seems to be working
Comment 2 Nick Clemens (kidclamp) 2020-02-26 15:53:44 UTC
Ah, a search like:
ti:chess AND kw:chess

does fail, because it translates to:
ti:chess ANDchess
Comment 3 Nick Clemens (kidclamp) 2020-10-27 11:31:36 UTC
Created attachment 112562 [details] [review]
Bug 24567: Don't strip spaces along with dangling colons

This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space

To test:
 1 - Have Koha using ES
 2 - Search for:
    ti:chess AND chess
 3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
 4 - Search for:
    ti:chess AND kw:chess
 5 - No result
 6 - Enable  DumpTemplateVarsIntranet  and  DumpSearchQueryTemplate
 7 - Repeate search and check page source
 8 - search_query has:
    title:chess ANDchess
 9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
     title:chess AND chess
Comment 4 Séverine Queune 2020-10-27 12:30:09 UTC
Created attachment 112564 [details] [review]
Bug 24567: Don't strip spaces along with dangling colons

This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space

To test:
 1 - Have Koha using ES
 2 - Search for:
    ti:chess AND chess
 3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
 4 - Search for:
    ti:chess AND kw:chess
 5 - No result
 6 - Enable  DumpTemplateVarsIntranet  and  DumpSearchQueryTemplate
 7 - Repeate search and check page source
 8 - search_query has:
    title:chess ANDchess
 9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
     title:chess AND chess

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Comment 5 Katrin Fischer 2020-11-01 13:18:52 UTC
 FAIL	Koha/SearchEngine/Elasticsearch/QueryBuilder.pm
   OK	  critic
   OK	  forbidden patterns
   OK	  git manipulation
   OK	  pod
   OK	  pod coverage
   OK	  spelling
   FAIL	  valid
		\3 better written as $3 
		\4 better written as $4 

Can you please check?
Comment 6 Nick Clemens (kidclamp) 2020-11-02 13:44:38 UTC
Created attachment 112832 [details] [review]
Bug 24567: (follow-up) Use dollar sign to refer to captures
Comment 7 Joonas Kylmälä 2020-11-03 09:54:44 UTC
Created attachment 112903 [details] [review]
Bug 24567: Don't strip spaces along with dangling colons

This updates the regex used for removing colons to capture those with space on either side, and remove the colon
while preserving the space

To test:
 1 - Have Koha using ES
 2 - Search for:
    ti:chess AND chess
 3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
 4 - Search for:
    ti:chess AND kw:chess
 5 - No result
 6 - Enable  DumpTemplateVarsIntranet  and  DumpSearchQueryTemplate
 7 - Repeate search and check page source
 8 - search_query has:
    title:chess ANDchess
 9 - Apply patch
10 - Repeat
11 - Seaerch works!
12 - query is now:
     title:chess AND chess

Signed-off-by: Séverine QUEUNE <severine.queune@bulac.fr>
Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 8 Joonas Kylmälä 2020-11-03 09:54:47 UTC
Created attachment 112904 [details] [review]
Bug 24567: (follow-up) Use dollar sign to refer to captures

Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 9 Joonas Kylmälä 2020-11-03 09:58:11 UTC
I only now noticed the warning

> Use of uninitialized value $3 in concatenation (.) or string at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 943.

Moving back to FQA until sure what to do with this.
Comment 10 Joonas Kylmälä 2020-11-03 10:35:56 UTC
Created attachment 112906 [details] [review]
Bug 24567: (QA follow-up) Remove warning in regex

Without this patch we get

Use of uninitialized value $3 in concatenation (.) or string at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/QueryBuilder.pm line 943.

This converts the | OR operator to two different regexes so that the
capture group variables will be defined in every case.

Signed-off-by: Joonas Kylmälä <joonas.kylmala@helsinki.fi>
Comment 11 Joonas Kylmälä 2020-11-03 10:36:40 UTC
Added follow-up to remove the warning. Passing QA.
Comment 12 Nick Clemens (kidclamp) 2020-11-04 16:07:42 UTC
*** Bug 25263 has been marked as a duplicate of this bug. ***
Comment 13 Jonathan Druart 2020-11-04 16:33:52 UTC
Pushed to master for 20.11, thanks to everybody involved!
Comment 14 Nick Clemens (kidclamp) 2021-03-12 13:24:03 UTC
*** Bug 24372 has been marked as a duplicate of this bug. ***
Comment 15 Christopher Brannon 2021-03-12 15:19:11 UTC
Can this be reproduced for 20.05?
Comment 16 Andrew Fuerste-Henry 2021-03-23 14:40:05 UTC
Pushed to 20.05.x for 20.05.10
Comment 17 Victor Grousset/tuxayo 2021-03-23 18:29:00 UTC
Not backported to oldoldstable (19.11.x). Feel free to ask if it's needed.
Comment 18 Martin Renvoize (ashimema) 2021-04-27 08:59:37 UTC
Any chance of getting this for 19.11 Victor?  It looks a reasonably simple patch to me, and we have a couple of customers enquiring about it :)
Comment 19 Victor Grousset/tuxayo 2021-04-27 17:48:17 UTC
Should the patch "Use dollar sign to refer to captures" be pushed to master?
Comment 20 Victor Grousset/tuxayo 2021-04-27 19:42:30 UTC
@Martin is that actually needed for 19.11.x?

     1 - Have Koha using ES
     2 - Search for:
        ti:chess AND chess
     3 - You should get a result in sample data, otherwise replace 'chess' with a title in your catalogue
     4 - Search for:
        ti:chess AND kw:chess
     5 - No result

I actually have results.
19.11.17, koha-testing-docker, marc21, ES enabled