Bug 28326 - If ElasticSearch enable is not possible to edit or save records with ' ES reserved charset' like []
Summary: If ElasticSearch enable is not possible to edit or save records with ' ES re...
Status: RESOLVED DUPLICATE of bug 28484
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: unspecified
Hardware: All All
: P5 - low major (vote)
Assignee: Nick Clemens (kidclamp)
QA Contact:
URL:
Keywords:
Depends on: 26313
Blocks:
  Show dependency treegraph
 
Reported: 2021-05-12 06:34 UTC by Hugo Agud
Modified: 2021-09-01 15:08 UTC (History)
7 users (show)

See Also:
Change sponsored?: ---
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
Version(s) released in:


Attachments
Bug 28326: Escape special characters in title when searching for analytics (2.11 KB, patch)
2021-05-14 13:35 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review
Bug 28326: Escape special characters in title when searching for analytics (2.16 KB, patch)
2021-05-14 21:25 UTC, David Nind
Details | Diff | Splinter Review
Bug 28326: Escape special characters in title when searching for analytics (1.77 KB, patch)
2021-05-17 10:23 UTC, Martin Renvoize
Details | Diff | Splinter Review
Bug 28326: Add escape_query function to deal with reserved characters (4.81 KB, patch)
2021-05-26 15:11 UTC, Nick Clemens (kidclamp)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Hugo Agud 2021-05-12 06:34:08 UTC
With ElasticSearch enabled it is not possible to create or edit any record with special charset in the title like []

[2021/05/12 06:29:49] [WARN] [Request] ** [hxxxxxx]-[400] [query_shard_exception] Failed to parse query [(host-item:[Romanza appassionata])], with: {"index_uuid":"VlZbj1MfQtWjWgdlnkSUXA","index":"koha_cxxx_biblios"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/share/koha/lib/Koha/SearchEngine/Elasticsearch/Search.pm line 96. With vars: {'status_code' => 400,'body' => {'error' => {'grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'root_cause' => [{'reason' => 'Failed to parse query [(host-item:[Romanza appassionata])]','index_uuid' => 'VlZbj1MfQtWjWgdlnkSUXA','index' => 'koha_xxxx_biblios','type' => 'query_shard_exception'}],'reason' => 'all shards failed','failed_shards' => [{'reason' => {'index_uuid' => 'VlZbj1MfQtWjWgdlnkSUXA','caused_by' => {'type' => 'parse_exception','caused_by' => {'type' => 'parse_exception','reason' => 'Encountered " <RANGE_GOOP> "appassionata "" at line 1, column 20.
Was expecting:
    "TO" ...
    '},'reason' => 'Cannot parse \'(host-item:[Romanza appassionata])\': Encountered " <RANGE_GOOP> "appassionata "" at line 1, column 20.
Was expecting:
    "TO" ...
Comment 1 Lucy Vaux-Harvey 2021-05-13 08:27:41 UTC
This is causing problems for one of our libraries, it would be great to see a fix for this.
Comment 2 Nick Clemens (kidclamp) 2021-05-14 13:35:09 UTC
Created attachment 120971 [details] [review]
Bug 28326: Escape special characters in title when searching for analytics

Bug 26313 introduced a search for analytics, which is great, but it started
sending titles to ES. This, unfrotunately, caused problems where the title
contained reserved ES characters.

As these searches should always be literl title searches, we should escape all the ES
characters before the search.

We should not make this a standard, however, as then it prevents using the characters for advanced ES
searching

To test:
1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch'
2 - Edit a record and add some reserved ES characters to the title
    https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
    e.g. De [Crasi] Aristophea.
3 - You get an error on the detail page loading
4 - Apply patch
5 - Reload
6 - The page successfully loads
Comment 3 David Nind 2021-05-14 21:25:42 UTC
Created attachment 121000 [details] [review]
Bug 28326: Escape special characters in title when searching for analytics

Bug 26313 introduced a search for analytics, which is great, but it started
sending titles to ES. This, unfrotunately, caused problems where the title
contained reserved ES characters.

As these searches should always be literl title searches, we should escape all the ES
characters before the search.

We should not make this a standard, however, as then it prevents using the characters for advanced ES
searching

To test:
1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch'
2 - Edit a record and add some reserved ES characters to the title
    https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
    e.g. De [Crasi] Aristophea.
3 - You get an error on the detail page loading
4 - Apply patch
5 - Reload
6 - The page successfully loads

Signed-off-by: David Nind <david@davidnind.com>
Comment 4 Martin Renvoize 2021-05-17 10:23:56 UTC
Created attachment 121035 [details] [review]
Bug 28326: Escape special characters in title when searching for analytics

Bug 26313 introduced a search for analytics, which is great, but it started
sending titles to ES. This, unfrotunately, caused problems where the title
contained reserved ES characters.

As these searches should always be literl title searches, we should escape all the ES
characters before the search.

We should not make this a standard, however, as then it prevents using the characters for advanced ES
searching

To test:
1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch'
2 - Edit a record and add some reserved ES characters to the title
    https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
    e.g. De [Crasi] Aristophea.
3 - You get an error on the detail page loading
4 - Apply patch
5 - Reload
6 - The page successfully loads

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com>
Comment 5 Martin Renvoize 2021-05-17 10:25:04 UTC
This gets us out of a bind, and works perfectly.  We should probably, in the future, move this into the QueryBuilder module, but I think that needs more thought and can be saved for the future.

Passing QA.
Comment 6 Jonathan Druart 2021-05-19 12:50:28 UTC
We need this code moved to a method and covered by tests, for re-usability.
Comment 7 Nick Clemens (kidclamp) 2021-05-26 15:11:50 UTC
Created attachment 121436 [details] [review]
Bug 28326: Add escape_query function to deal with reserved characters

Bug 26313 introduced a search for analytics, which is great, but it started
sending titles to ES. This, unfortunately, caused problems where the title
contained reserved ES characters.

As these searches should always be literal title searches, we should escape all the ES
characters before the search.

We should not make this a standard, however, as then it prevents using the characters for advanced ES
searching, but we still add a function for future reuse

To test:
1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch'
2 - Edit a record and add some reserved ES characters to the title
    https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html
    e.g. De [Crasi] Aristophea.
3 - You get an error on the detail page loading
4 - Apply patch
5 - Reload
6 - The page successfully loads
7 - prove -v t/db_dependent/Koha/SearchEngine/Search.t
Comment 8 Victor Grousset/tuxayo 2021-05-26 19:25:45 UTC Comment hidden (obsolete)
Comment 9 Victor Grousset/tuxayo 2021-05-27 03:13:23 UTC
More tests. The following name cause the crash despite the patch.

== Tested strings, work ==
quikc~ brwn~ foks~
name:/joh?n(ath[oa]n)/
count:[10 TO *]
age:(+>=10 +<20)
quick^2 fox
The reserved characters are: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /


== Crash ==
"fox quick"~5
tag:{alpha TO omega}
Nmap network scanning :  <> ><
Comment 10 Martin Renvoize 2021-09-01 15:07:09 UTC
I think this is now superseded by bug 28316

*** This bug has been marked as a duplicate of bug 28316 ***
Comment 11 Martin Renvoize 2021-09-01 15:07:44 UTC

*** This bug has been marked as a duplicate of bug 28484 ***
Comment 12 Martin Renvoize 2021-09-01 15:08:20 UTC
Sorry.. bug 28484 which uses the work from bug 28316