Bug 27252

Summary: ES5 no longer supported (since 20.11.00)
Product: Koha Reporter: Jonathan Druart <jonathan.druart>
Component: Architecture, internals, and plumbingAssignee: Nick Clemens (kidclamp) <nick>
Status: CLOSED FIXED QA Contact: Tomás Cohen Arazi <tomascohen>
Severity: major    
Priority: P5 - low CC: andrewfh, caroline.cyr-la-rose, fridolin.somers, mtj, tomascohen, victor
Version: MainKeywords: Manual, rel_20_11_candidate
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=27770
Change sponsored?: --- Patch complexity: ---
Documentation contact: Caroline Cyr La Rose Documentation submission: https://gitlab.com/koha-community/koha-manual/-/merge_requests/575
Text to go in the release notes:
This prepares Koha to officially no longer support Elasticsearch 5.X. It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not to enable this feature.
Version(s) released in:
21.05.00,20.11.02,20.05.07,19.11.14
Bug Depends on: 27070    
Bug Blocks: 27448    
Attachments: Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info
Bug 27252: [20.05 and earlier] Default to cross fields off
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info
[20.05 and earlier] Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info
[20.05 and earlier] Bug 27252: Default to cross fields off

Description Jonathan Druart 2020-12-16 13:15:06 UTC
Since 20.11.00 and bug 27070 we dropped the support for ES5.
We should add a warning on the about page.
Comment 1 Nick Clemens (kidclamp) 2020-12-16 19:36:52 UTC
Created attachment 114466 [details] [review]
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info

This patch prepares Koha to officially no longer support Elasticsearch 5.X

It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature

It updates the about page to add a deprecation warning if a site is running ES5

To test:
 1 - Be running Koha with Elasticsearch 5.X
 2 - Attempt to search
   Error: Unable to perform your search. Please try again.
 3 - Apply patch
 4 - Update database
 5 - Searching works
 6 - Find syspref 'ElasticsearchCrossFields'
 7 - Enable it
 8 - Searching is now broken
 9 - Check the about page
 10 - you can now see the Elasticsearch version
 11 - The systeminformation tab has a deprectaion warning
 12 - Set SearchEngine preference to 'Zebra'
 13 - View the about page - no warnings
 14 - Test again with ES6 - searching should "work" with either pref setting
 15 - There should be no warning on about pages
Comment 2 Jonathan Druart 2020-12-17 13:48:49 UTC
Shouldn't we turn the feature on for master and 20.11?
Comment 3 Victor Grousset/tuxayo 2020-12-17 20:44:04 UTC
It's seems that all the CI jobs are using ES6 so the tests should fail with this off by default.

prove -r t/Koha/SearchEngine/Elasticsearch* t/db_dependent/Koha_Elasticsearch.t t/Koha_SearchEngine_Elasticsearch_Browse.t t/db_dependent/Koha/SearchEngine/

It turns out not :o
Comment 4 Jonathan Druart 2020-12-18 09:23:33 UTC
Bug 27070 did not add tests so I don't know which tests would be supposed to fail.
Comment 5 Victor Grousset/tuxayo 2020-12-18 21:51:01 UTC
I thought searching with ES6 would be broken by default. And thus that some tests  would fail. Because the syspref would allow to choose to have ES 6 working or ES 5 working and would default to ES 5.
Comment 6 Nick Clemens (kidclamp) 2020-12-21 11:46:23 UTC
(In reply to Victor Grousset/tuxayo from comment #5)
> I thought searching with ES6 would be broken by default. And thus that some
> tests  would fail. Because the syspref would allow to choose to have ES 6
> working or ES 5 working and would default to ES 5.

After bug 27070 ES5 searching was broken. This patch retores the ability to use ES5 - since Jenkins was using 6 everywhere, there was no problem
Comment 7 Victor Grousset/tuxayo 2020-12-21 13:21:49 UTC
Alright, I missed «with ES6 - searching should "work" with either pref setting»

Thanks for the explanations.
Comment 8 Victor Grousset/tuxayo 2020-12-21 13:43:22 UTC
Created attachment 114553 [details] [review]
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info

This patch prepares Koha to officially no longer support Elasticsearch 5.X

It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature

It updates the about page to add a deprecation warning if a site is running ES5

To test:
 1 - Be running Koha with Elasticsearch 5.X
 2 - Attempt to search
   Error: Unable to perform your search. Please try again.
 3 - Apply patch
 4 - Update database
 5 - Searching works
 6 - Find syspref 'ElasticsearchCrossFields'
 7 - Enable it
 8 - Searching is now broken
 9 - Check the about page
 10 - you can now see the Elasticsearch version
 11 - The systeminformation tab has a deprectaion warning
 12 - Set SearchEngine preference to 'Zebra'
 13 - View the about page - no warnings
 14 - Test again with ES6 - searching should "work" with either pref setting
 15 - There should be no warning on about pages

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 9 Victor Grousset/tuxayo 2020-12-21 13:45:43 UTC
Nice changes thanks :)

Should it be recommended to enable ElasticsearchCrossFields when using ES 6? To not have bug 27070
Comment 10 Jonathan Druart 2020-12-21 14:25:54 UTC
(In reply to Victor Grousset/tuxayo from comment #9)
> Should it be recommended to enable ElasticsearchCrossFields when using ES 6?
> To not have bug 27070

Yes, see comment 2. I think it should be turned on by default. Nick, what do you think?
Comment 11 Nick Clemens (kidclamp) 2020-12-23 14:31:13 UTC
Created attachment 114672 [details] [review]
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info

This patch prepares Koha to officially no longer support Elasticsearch 5.X

It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature

It updates the about page to add a deprecation warning if a site is running ES5

To test:
 1 - Be running Koha with Elasticsearch 5.X
 2 - Attempt to search
   Error: Unable to perform your search. Please try again.
 3 - Apply patch
 4 - Update database
 5 - Searching works
 6 - Find syspref 'ElasticsearchCrossFields'
 7 - Enable it
 8 - Searching is now broken
 9 - Check the about page
 10 - you can now see the Elasticsearch version
 11 - The systeminformation tab has a deprectaion warning
 12 - Set SearchEngine preference to 'Zebra'
 13 - View the about page - no warnings
 14 - Test again with ES6 - searching should "work" with either pref setting
 15 - There should be no warning on about pages

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Comment 12 Nick Clemens (kidclamp) 2020-12-23 14:31:16 UTC
Created attachment 114673 [details] [review]
Bug 27252: [20.05 and earlier] Default to cross fields off
Comment 13 Nick Clemens (kidclamp) 2020-12-23 14:31:38 UTC
(In reply to Jonathan Druart from comment #2)
> Shouldn't we turn the feature on for master and 20.11?

Yes, amended patch and attached followup for earlier versions
Comment 14 Tomás Cohen Arazi 2021-01-05 14:30:01 UTC
Created attachment 114878 [details] [review]
Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info

This patch prepares Koha to officially no longer support Elasticsearch 5.X

It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature

It updates the about page to add a deprecation warning if a site is running ES5

To test:
 1 - Be running Koha with Elasticsearch 5.X
 2 - Attempt to search
   Error: Unable to perform your search. Please try again.
 3 - Apply patch
 4 - Update database
 5 - Searching works
 6 - Find syspref 'ElasticsearchCrossFields'
 7 - Enable it
 8 - Searching is now broken
 9 - Check the about page
 10 - you can now see the Elasticsearch version
 11 - The systeminformation tab has a deprectaion warning
 12 - Set SearchEngine preference to 'Zebra'
 13 - View the about page - no warnings
 14 - Test again with ES6 - searching should "work" with either pref setting
 15 - There should be no warning on about pages

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 15 Tomás Cohen Arazi 2021-01-05 14:55:22 UTC
Created attachment 114881 [details] [review]
[20.05 and earlier] Bug 27252: Add deprecation warning, cross_fields pref, and add version to ES info

This patch prepares Koha to officially no longer support Elasticsearch 5.X

It adds a new system preference 'ElasticsearchCrossFields' to allow users to choose whether or not
to enable this feature

It updates the about page to add a deprecation warning if a site is running ES5

To test:
 1 - Be running Koha with Elasticsearch 5.X
 2 - Attempt to search
   Error: Unable to perform your search. Please try again.
 3 - Apply patch
 4 - Update database
 5 - Searching works
 6 - Find syspref 'ElasticsearchCrossFields'
 7 - Enable it
 8 - Searching is now broken
 9 - Check the about page
 10 - you can now see the Elasticsearch version
 11 - The systeminformation tab has a deprectaion warning
 12 - Set SearchEngine preference to 'Zebra'
 13 - View the about page - no warnings
 14 - Test again with ES6 - searching should "work" with either pref setting
 15 - There should be no warning on about pages

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 16 Tomás Cohen Arazi 2021-01-05 14:55:27 UTC
Created attachment 114882 [details] [review]
[20.05 and earlier] Bug 27252: Default to cross fields off

Signed-off-by: Tomas Cohen Arazi <tomascohen@theke.io>
Comment 17 Jonathan Druart 2021-01-07 14:37:50 UTC
Pushed to master for 21.05, thanks to everybody involved!
Comment 18 Fridolin Somers 2021-01-07 15:33:02 UTC
Pushed to 20.11.x for 20.11.02
Comment 19 Andrew Fuerste-Henry 2021-01-07 19:06:10 UTC
Pushed to 20.05.x for 20.05.07
Comment 20 Victor Grousset/tuxayo 2021-01-17 22:34:34 UTC
Backported: Pushed to 19.11.x branch for 19.11.14

After some testing, the syspref has the expected effect with ES 5 and 6

However I can't see the deprecation warning. With ES 5 and syspref SearchEngine = elasticsearch, the about page shows "Version: 5.6.16" but no warning in the System information tab.
Comment 21 Victor Grousset/tuxayo 2021-01-17 23:06:25 UTC
No warning in 20.05.x either but the warning is here in 20.11
Comment 22 Jonathan Druart 2021-01-18 09:59:31 UTC
(In reply to Victor Grousset/tuxayo from comment #21)
> No warning in 20.05.x either but the warning is here in 20.11

You need bug 19542.
Comment 23 Victor Grousset/tuxayo 2021-01-18 17:46:24 UTC
We have the commits in 19.11.x and 20.05.x , since 18.05.
The ES version info is here in the "Server information" tab of the about page.