Created attachment 148849 [details] [review]
Bug 33353: Add compatibility with Elasticsearch 8.x

Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html

Tested with Search::Elasticsearch 6.80 and 7.717

Test plan:
1. Install Elasticsearch 8
   (use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
   It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
   It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
   before

Created attachment 148850 [details] [review]
Bug 33353: Add compatibility with Search::Elasticsearch 8.0

For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:

{
    total => {
        value => 1,
        relation => 'gte'
    }
}

Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)

Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220

We might need to find an alternative for future versions of
Elasticsearch

lol replacing JSON::true by the magic number 1, weird change that ES are doing.

Created attachment 148937 [details] [review]
Bug 33353: Add compatibility with Elasticsearch 8.x

Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html

Tested with Search::Elasticsearch 6.80 and 7.717

Test plan:
1. Install Elasticsearch 8
   (use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
   It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
   It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
   before

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Created attachment 148938 [details] [review]
Bug 33353: Add compatibility with Search::Elasticsearch 8.0

For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:

{
    total => {
        value => 1,
        relation => 'gte'
    }
}

Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)

Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220

We might need to find an alternative for future versions of
Elasticsearch

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>

Anyway, it works, also with OpenSearch 2.x!

---

Here are all tests that to me seem relevant for ES/OS

prove -r t/Koha/SearchEngine/Elasticsearch* t/db_dependent/Koha_Elasticsearch.t t/Koha_SearchEngine_Elasticsearch_Browse.t t/db_dependent/Koha/SearchEngine/ t/db_dependent/Koha/BackgroundJob/UpdateElasticIndex.t

All pass.

I mentioned a warning:

t/Koha/SearchEngine/Elasticsearch/Search.t ..................... Name "Koha::SearchEngine::Elasticsearch::AUTHORITIES_INDEX" used only once: possible typo at t/Koha/SearchEngine/Elasticsearch/Search.t line 55.

But it happens with ES6.


---

Also, in about.pl it doesn't show ES8.
But it works with OpenSearch 2.x
And not with ES 7???? Weird

By the way, these patches mean end of compatibility with ES6.

Not a problem in my opinion, ES6 is EOL since 10 Feb 2022.
And now that we have OpenSearch 1.x compat (still need to officialize it but it's there) people that are bothered with the new ES license (since ES 7.11) not really being libre/open source anymore (it's a mess, very controversial and blurry) can use OpenSearch 1.x

Created attachment 148988 [details] [review]
Bug 33353: Add compatibility with Elasticsearch 8.x

Elasticsearch 8 removed the support of types. This patch adapts the
requests accordingly.
With this patch, Koha will still be compatible with Elasticsearch 7.x
but will no longer work with Elasticsearch 6.x
Since Elasticsearch 6.x is no longer maintained, this should not be a
problem.

https://www.elastic.co/guide/en/elasticsearch/reference/7.17/removal-of-types.html

Tested with Search::Elasticsearch 6.80 and 7.717

Test plan:
1. Install Elasticsearch 8
   (use docker image: docker.io/koha/elasticsearch-icu:8.x)
2. Change the elasticsearch server location in $KOHA_CONF
3. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
   It should fail
4. Apply the patch
5. Run misc/search_tools/rebuild_elasticsearch.pl -r -b -v
   It should end with "Total XXX records indexed"
6. Try to search some biblios and verify that it works the same as
   before

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>

Created attachment 148989 [details] [review]
Bug 33353: Add compatibility with Search::Elasticsearch 8.0

For some reasons, with Search::Elasticsearch 8.0 JSON::true is
interpreted as 1 and Elasticsearch will always respond that it found "at
least 1 record". Something like this:

{
    total => {
        value => 1,
        relation => 'gte'
    }
}

Replacing JSON::true by \1 works with every version of
Search::Elasticsearch I tested (6.80, 7.717, 8.0)

Also worth noting:
Search::Elasticsearch will stop being supported by Elastic
https://github.com/elastic/elasticsearch-perl/issues/220

We might need to find an alternative for future versions of
Elasticsearch

Signed-off-by: Victor Grousset/tuxayo <victor@tuxayo.net>
Signed-off-by: Philip Orr <philip.orr@lmscloud.de>

What documentation do we need to update once this is pushed? Wiki, website,...?

Wiki only I guess (which isn't up to date about ES 7 and OS 1 support ^^")

https://wiki.koha-community.org/wiki/System_requirements_and_recommendations

It's linked in every release notes so hopefully it's read on major releases. ES users are larger instances so hopefully they is more confidence it's read that and or have a test instance where they will quickly see the problem to run the wrong ES version.

By the way, thanks Philip for the test. It counts as a signoff and so I can QA! A second signoff is always useful.

Works, makes sense, QA script happy, code looks good, passing QA :)

After having this in main/master there will be the question of backporting (now or later) to 22.11 which is the LTS. Because before the end of the LTS it's likely that ES 7 won't be supported anymore.

Here https://www.elastic.co/support/eol#elasticsearch
we can see that branch 7.x will be supported until 9.x comes out.

There is no alpha version of 9.x for now, but when there will be, the stable release can be expected 6 months later. (learned this in ES forums) So people interested can keep an eye here: https://github.com/elastic/elasticsearch/tags
for when the 1st 9.x alpha pops.

Also, for OpenSearch, it's already planned that 1.x will reach end of life before the end of 22.11's support.
https://opensearch.org/releases.html => "Maintenance Window End" => "December 31, 2023"

So what's the best strategy? (because it implies dropping 6.x support)
1. backport now
2. backport when/a bit before OpenSearch 1.x support ends (December 31, 2023)
3. backport when/a bit before ElasticSearch 7.x support will have also ended. (we don't know yet, it could be sooner than OS 1 or later)
4. don't backport and require ES 7 or OS 1 Koha admins to keep an end of life search engine or upgrade from the LTS.

(In reply to Victor Grousset/tuxayo from comment #13)
> After having this in main/master there will be the question of backporting
> (now or later) to 22.11 which is the LTS. Because before the end of the LTS
> it's likely that ES 7 won't be supported anymore.
> 
> Here https://www.elastic.co/support/eol#elasticsearch
> we can see that branch 7.x will be supported until 9.x comes out.
> 
> There is no alpha version of 9.x for now, but when there will be, the stable
> release can be expected 6 months later. (learned this in ES forums) So
> people interested can keep an eye here:
> https://github.com/elastic/elasticsearch/tags
> for when the 1st 9.x alpha pops.
> 
> Also, for OpenSearch, it's already planned that 1.x will reach end of life
> before the end of 22.11's support.
> https://opensearch.org/releases.html => "Maintenance Window End" =>
> "December 31, 2023"
> 
> So what's the best strategy? (because it implies dropping 6.x support)
> 1. backport now
> 2. backport when/a bit before OpenSearch 1.x support ends (December 31, 2023)
> 3. backport when/a bit before ElasticSearch 7.x support will have also
> ended. (we don't know yet, it could be sooner than OS 1 or later)
> 4. don't backport and require ES 7 or OS 1 Koha admins to keep an end of
> life search engine or upgrade from the LTS.

This bug caught my attention last week, and I have been watching with interest to see what responses there would be to Victor's question here. In the absence of any (so far), I would like to offer the perspective of a system administrator in the wild:

- The cataloger for our consortium of small libraries has been itching to implement Elasticsearch for more than a year.
- In examining the request, I was disconcerted by a few facts about Elasticsearch: 1. The version included by Koha appeared to be one that had reached end-of-life with the publisher, 2. Even the supported version had been associated with significant data breaches, and 3. There seemed to be a lack of clarity following the change in licensing and the open source world's response to it.
- Given the above, there was no scenario in this corner of the multiverse in which I would have installed Elasticsearch on the internet-facing servers for which I was responsible.
- HOWEVER, seeing OpenSearch on the horizon creates a new situation. Being well-supported by AWS, it has a chance of being a viable and safe alternative.
- Our production server runs 22.05, BUT I can imagine a willingness to leap forward a bit if OpenSearch is backported to 22.11.
- If that is indeed the future of more advanced indexing and searching, then it seems logical (to me) to backport it to an LTS release, if you have the resources to do so.

Perhaps the question needs the attention of a broader or more representative forum, if you have one.

> This bug caught my attention last week, and I have been watching with interest to see what responses there would be to Victor's question here. In the absence of any (so far), I would like to offer the perspective of a system administrator in the wild:

Hi, welcome :) More input can definitely help to unblock this.

> 1. The version included by Koha appeared to be one that had reached end-of-life with the publisher

ES 7 and OpenSearch 1.X (which isn't eol) has been supported since a year. But we didn't update the systems requirements documentation (now it's finally updated) due to not having the time to search for real work usage to confidently claim support :-/

> 2. Even the supported version had been associated with significant data breaches

You mean the log4j vulnerabilities? (latest 6.x patched them) or like bad defaults?
Is your statement still valid for ES 7.x?

> 3. There seemed to be a lack of clarity following the change in licensing and the open source world's response to it.

Indeed! About that, to have more material to raise awareness about the issue, do you happen to know more about the open source world's response to it? Besides Debian, Fedora and the Open Source Initiative not considering the SSPL license libre/open source?

> - Our production server runs 22.05, BUT I can imagine a willingness to leap forward a bit if OpenSearch is backported to 22.11.

You can do that right away with OpenSearch 1.x , 22.05 supported it on launch. Sorry again for the delay in advertising it's support.

(In reply to Victor Grousset/tuxayo from comment #15)

Hi, Victor, and thank you for following up!

> > 1. The version included by Koha appeared to be one that had reached end-of-life with the publisher
> 
> ES 7 and OpenSearch 1.X (which isn't eol) has been supported since a year.
> But we didn't update the systems requirements documentation (now it's
> finally updated) due to not having the time to search for real work usage to
> confidently claim support :-/

Our Koha installations represent a fraction of the systems that I support or manage in my role. I have to judiciously balance the time and effort I give to any single system. When considering the addition of a new software, I only do so when that software is actively developed by the publisher, is clearly supported on the target system, and has a well-documented installation/integration process. Since I didn't see that with Elasticsearch and Koha, I avoided it.

> > 2. Even the supported version had been associated with significant data breaches
> 
> You mean the log4j vulnerabilities? (latest 6.x patched them) or like bad
> defaults?
> Is your statement still valid for ES 7.x?

No, I believe log4j was a separate issue. The data breaches were in the news a few years ago. Here's an example:
https://www.techradar.com/news/what-is-elasticsearch-and-why-is-it-involved-in-so-many-data-leaks
That article doesn't specify which CVEs were involved, but I wasn't about to create a new installation with an old, unsupported (read: perhaps unpatched) version that could _possibly_ make our system vulnerable to the exfiltration of patron and staff data. That would be irresponsible, especially since some portion of those people are under the protection of the EU GDPR.

> > 3. There seemed to be a lack of clarity following the change in licensing and the open source world's response to it.
> 
> Indeed! About that, to have more material to raise awareness about the
> issue, do you happen to know more about the open source world's response to
> it? Besides Debian, Fedora and the Open Source Initiative not considering
> the SSPL license libre/open source?

The details of whether or not ES truly can be considered 'open source' are not actually important to me. Observing a general lack of clarity, and not knowing the Koha development team's point of view on the matter, I couldn't be certain of the future of ES with regards to Koha. In my situation, that caused me to recommend against installing ES.

> > - Our production server runs 22.05, BUT I can imagine a willingness to leap forward a bit if OpenSearch is backported to 22.11.
> 
> You can do that right away with OpenSearch 1.x , 22.05 supported it on
> launch. Sorry again for the delay in advertising it's support.

That's why I was pleased to see this entry and the references to OpenSearch. Once I see an open declaration of support for its usage, and once I see some clear instructions for integrating it, I will be willing to install it on our test server. I have to approach the matter cautiously to protect the correct function of our systems – as well as my time and sanity.

Thanks again for your response! I just want to let you all have an idea how one system administrator views the situation.

I think before this is pushed we need a way for those using ES7 to remove the type parameter, and continue to use Koha.

With these patches, the upgrade would require:
Upgrade Koha
Upgrade ES
Reindex ES


I'd like to see a path of:
Reindex Koha on ES 7 to remove type
then either:
Upgrade Koha
Upgrade ES

Making the upgrade of Koha force the ES upgrade without allowing it before feels like it could cause issues

I am removing my block here - while I suspect we are going to have some difficulty with upgrades I would rather see us move forward than be stuck. 

I believe you can use es directly to update and remove types and avoid reindex:

curl -X POST http://es:9200/_reindex -H 'Content-Type: application/json' -d '{"source":{"index": "koha_kohadev_biblios","type": "data"},"dest":{"index": "koha_kohadev_biblio_s"}}'

that reindexes ES internally to remove the types - it does require renaming the index so would need some sort of alias or something, but it would be nice if, on another report, we could provide a tool for this update

(In reply to Nick Clemens from comment #17)
> Making the upgrade of Koha force the ES upgrade without allowing it before
> feels like it could cause issues

Thanks for catching that ﷐[U+1F631]﷑ that's a big miss in my review. Requiring reindex wouldn't make the upgrade as simple as hoped.

How can one reproduce an ES upgrade with KTD? That would mean shutting down the ES7 container and restart an ES8 one without loosing the existing data.
https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html
Maybe something with a snapshot and moving the snapshot files out of the container and putting then in the ES8 container?
Or starting a parallel ES8 container, having it replicate the data from the ES7 one and replace the ES7 with an ES8 (to have the right container name to plug in KTD) and replicate again?

(In reply to Nick Clemens from comment #18)
> I believe you can use es directly to update and remove types and avoid
> reindex:
> 
> curl -X POST http://es:9200/_reindex -H 'Content-Type: application/json' -d
> '{"source":{"index": "koha_kohadev_biblios","type": "data"},"dest":{"index":
> "koha_kohadev_biblio_s"}}'
> 
> that reindexes ES internally to remove the types

So it's much faster than reindexing from Koha's data, that's the gain, right?

>  - it does require renaming
> the index so would need some sort of alias or something

Does doing the rename again would allow to get back to original name and have no config to touch?

(In reply to Victor Grousset/tuxayo from comment #19)
> (In reply to Nick Clemens from comment #17)
> > Making the upgrade of Koha force the ES upgrade without allowing it before
> > feels like it could cause issues
> 
> Thanks for catching that ﷐[U+1F631]﷑ that's a big miss in my review. Requiring
> reindex wouldn't make the upgrade as simple as hoped.
> 
> How can one reproduce an ES upgrade with KTD? That would mean shutting down
> the ES7 container and restart an ES8 one without loosing the existing data.
> https://www.elastic.co/guide/en/cloud/current/ec-migrating-data.html
> Maybe something with a snapshot and moving the snapshot files out of the
> container and putting then in the ES8 container?
> Or starting a parallel ES8 container, having it replicate the data from the
> ES7 one and replace the ES7 with an ES8 (to have the right container name to
> plug in KTD) and replicate again?

That would be tricky - maybe if you setup a volume, and then just connected new container to the volume? Or get into the container and perform an upgrade directly?
I think it should work for testing to spin up on ES7 - apply these patches, ES won't work, then do the reindex (as below or via Koha) and confirm that it works. We aren't removing 7 support afaik, but require 7 or higher after this I believe

> 
> (In reply to Nick Clemens from comment #18)
> > I believe you can use es directly to update and remove types and avoid
> > reindex:
> So it's much faster than reindexing from Koha's data, that's the gain, right?

Yes, as I understand it moves the data to remove the type level, so we don't have to cycle through every record and parse into ES - I tested at Hackfest and it was exponentially faster

> 
> >  - it does require renaming
> > the index so would need some sort of alias or something
> 
> Does doing the rename again would allow to get back to original name and
> have no config to touch?

I think you either setup an alias with the old that points to the new name - or do the reindex, delete the old one, then rename the new one

Pushed to master for 23.11.

Nice work everyone, thanks!

(In reply to Nick Clemens from comment #20)
> (In reply to Victor Grousset/tuxayo from comment #19)
> > Does doing the rename again would allow to get back to original name and
> > have no config to touch?
> 
> I think you either setup an alias with the old that points to the new name -
> or do the reindex, delete the old one, then rename the new one

Great, the 2nd possibility was what I was hopping to be possible.

(for anyone confused, reindex here mean internal reindex, Comment 18)

So as it stands what to know about upgrading, now that it's in main/master is that people will need either do a full reindex. Or an internal reindex, details in comment 18 and the end of comment 20

> people will need either do a full reindex. Or an internal reindex

Also, the end of ES 6 support! Provisional 23.11 section created here: https://wiki.koha-community.org/wiki/System_requirements_and_recommendations

----

(In reply to Nick Clemens from comment #20)
> (In reply to Victor Grousset/tuxayo from comment #19)
> > How can one reproduce an ES upgrade with KTD? 
>
> That would be tricky - maybe if you setup a volume, and then just connected
> new container to the volume? Or get into the container and perform an
> upgrade directly?
> I think it should work for testing to spin up on ES7 - apply these patches,
> ES won't work, then do the reindex (as below or via Koha) and confirm that
> it works. We aren't removing 7 support afaik, but require 7 or higher after
> this I believe

Thanks, I'll keep that noted :)

Title and release notes text completed. Feel free to amend if needed.

Enhancement not pushed to 23.05.x

Yep, as of now, ES 7.x and OS 1.x don't have a planned EOL. So no need for now to force installs still with ES 6.x (even if it's EOL since some time) to upgrade in minor Koha release.

The question would only come back when the old Koha releases are going to end up without support of a maintained ES or OS. So a long as ES 7.x and OS 1.x don't have a planned EOL, no worries.