This is causing problems for one of our libraries, it would be great to see a fix for this. Created attachment 120971 [details] [review] Bug 28326: Escape special characters in title when searching for analytics Bug 26313 introduced a search for analytics, which is great, but it started sending titles to ES. This, unfrotunately, caused problems where the title contained reserved ES characters. As these searches should always be literl title searches, we should escape all the ES characters before the search. We should not make this a standard, however, as then it prevents using the characters for advanced ES searching To test: 1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch' 2 - Edit a record and add some reserved ES characters to the title https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html e.g. De [Crasi] Aristophea. 3 - You get an error on the detail page loading 4 - Apply patch 5 - Reload 6 - The page successfully loads Created attachment 121000 [details] [review] Bug 28326: Escape special characters in title when searching for analytics Bug 26313 introduced a search for analytics, which is great, but it started sending titles to ES. This, unfrotunately, caused problems where the title contained reserved ES characters. As these searches should always be literl title searches, we should escape all the ES characters before the search. We should not make this a standard, however, as then it prevents using the characters for advanced ES searching To test: 1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch' 2 - Edit a record and add some reserved ES characters to the title https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html e.g. De [Crasi] Aristophea. 3 - You get an error on the detail page loading 4 - Apply patch 5 - Reload 6 - The page successfully loads Signed-off-by: David Nind <david@davidnind.com> Created attachment 121035 [details] [review] Bug 28326: Escape special characters in title when searching for analytics Bug 26313 introduced a search for analytics, which is great, but it started sending titles to ES. This, unfrotunately, caused problems where the title contained reserved ES characters. As these searches should always be literl title searches, we should escape all the ES characters before the search. We should not make this a standard, however, as then it prevents using the characters for advanced ES searching To test: 1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch' 2 - Edit a record and add some reserved ES characters to the title https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html e.g. De [Crasi] Aristophea. 3 - You get an error on the detail page loading 4 - Apply patch 5 - Reload 6 - The page successfully loads Signed-off-by: David Nind <david@davidnind.com> Signed-off-by: Martin Renvoize <martin.renvoize@ptfs-europe.com> This gets us out of a bind, and works perfectly. We should probably, in the future, move this into the QueryBuilder module, but I think that needs more thought and can be saved for the future. Passing QA. We need this code moved to a method and covered by tests, for re-usability. Created attachment 121436 [details] [review] Bug 28326: Add escape_query function to deal with reserved characters Bug 26313 introduced a search for analytics, which is great, but it started sending titles to ES. This, unfortunately, caused problems where the title contained reserved ES characters. As these searches should always be literal title searches, we should escape all the ES characters before the search. We should not make this a standard, however, as then it prevents using the characters for advanced ES searching, but we still add a function for future reuse To test: 1 - Have ES setup and running and swtich SearchEngine syspref to 'Elasticsearch' 2 - Edit a record and add some reserved ES characters to the title https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-query-string-query.html e.g. De [Crasi] Aristophea. 3 - You get an error on the detail page loading 4 - Apply patch 5 - Reload 6 - The page successfully loads 7 - prove -v t/db_dependent/Koha/SearchEngine/Search.t The following name cause the crash despite the patch
tag:{alpha TO omega}
More tests. The following name cause the crash despite the patch.
== Tested strings, work ==
quikc~ brwn~ foks~
name:/joh?n(ath[oa]n)/
count:[10 TO *]
age:(+>=10 +<20)
quick^2 fox
The reserved characters are: + - = && || > < ! ( ) { } [ ] ^ " ~ * ? : \ /
== Crash ==
"fox quick"~5
tag:{alpha TO omega}
Nmap network scanning : <> ><
I think this is now superseded by bug 28316 *** This bug has been marked as a duplicate of bug 28316 *** *** This bug has been marked as a duplicate of bug 28484 *** |
With ElasticSearch enabled it is not possible to create or edit any record with special charset in the title like [] [2021/05/12 06:29:49] [WARN] [Request] ** [hxxxxxx]-[400] [query_shard_exception] Failed to parse query [(host-item:[Romanza appassionata])], with: {"index_uuid":"VlZbj1MfQtWjWgdlnkSUXA","index":"koha_cxxx_biblios"}, called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /usr/share/koha/lib/Koha/SearchEngine/Elasticsearch/Search.pm line 96. With vars: {'status_code' => 400,'body' => {'error' => {'grouped' => bless( do{\(my $o = 1)}, 'JSON::PP::Boolean' ),'root_cause' => [{'reason' => 'Failed to parse query [(host-item:[Romanza appassionata])]','index_uuid' => 'VlZbj1MfQtWjWgdlnkSUXA','index' => 'koha_xxxx_biblios','type' => 'query_shard_exception'}],'reason' => 'all shards failed','failed_shards' => [{'reason' => {'index_uuid' => 'VlZbj1MfQtWjWgdlnkSUXA','caused_by' => {'type' => 'parse_exception','caused_by' => {'type' => 'parse_exception','reason' => 'Encountered " <RANGE_GOOP> "appassionata "" at line 1, column 20. Was expecting: "TO" ... '},'reason' => 'Cannot parse \'(host-item:[Romanza appassionata])\': Encountered " <RANGE_GOOP> "appassionata "" at line 1, column 20. Was expecting: "TO" ...