Bug 38345 - Restore support for OpenSearch
Summary: Restore support for OpenSearch
Status: Pushed to stable
Alias: None
Product: Koha
Classification: Unclassified
Component: Searching - Elasticsearch (show other bugs)
Version: Main
Hardware: All All
: P5 - low normal
Assignee: Jonathan Druart
QA Contact: Martin Renvoize (ashimema)
URL: https://gitlab.com/koha-community/koh...
Keywords:
Depends on:
Blocks:
 
Reported: 2024-11-04 14:38 UTC by Tomás Cohen Arazi (tcohen)
Modified: 2026-01-16 16:17 UTC (History)
12 users (show)

See Also:
GIT URL:
Initiative type: ---
Sponsorship status: ---
Comma delimited list of Sponsors:
Crowdfunding goal: 0
Patch complexity: ---
Documentation contact:
Documentation submission:
Text to go in the release notes:
This fixes Koha so that OpenSearch now works, and is now covered again by continuous integration tests (these have been failing for some time in Jenkins). Note: See the test plan for guidance on how to start KTD successfully with OS1 and OS2 if you have issues.
Version(s) released in:
26.05.00,25.11.01
Circulation function:


Attachments
Bug 38345: Restore support for OpenSearch (2.95 KB, patch)
2025-12-15 14:48 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 38345: Restore support for OpenSearch (2.88 KB, patch)
2025-12-16 08:51 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 38345: Remove warnings (1.01 KB, patch)
2025-12-16 11:38 UTC, Jonathan Druart
Details | Diff | Splinter Review
Bug 38345: Restore support for OpenSearch (2.92 KB, patch)
2025-12-16 18:11 UTC, David Nind
Details | Diff | Splinter Review
Bug 38345: Remove warnings (1.05 KB, patch)
2025-12-16 18:11 UTC, David Nind
Details | Diff | Splinter Review
Bug 38345: Restore support for OpenSearch (2.99 KB, patch)
2025-12-17 13:57 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review
Bug 38345: Remove warnings (1.12 KB, patch)
2025-12-17 13:57 UTC, Martin Renvoize (ashimema)
Details | Diff | Splinter Review

Note You need to log in before you can comment on or make changes to this bug.
Description Tomás Cohen Arazi (tcohen) 2024-11-04 14:38:58 UTC
Truth is our test suite has been exploding consistently when trying to launch OpenSearch:

* https://jenkins.koha-community.org/job/Koha_Main_OS2/
* https://jenkins.koha-community.org/job/Koha_Main_OS1/

This has some precedent discussion on [issue 429](https://gitlab.com/koha-community/koha-testing-docker/-/issues/429) in `KTD` project issues.

We are not going to remove it, I guess. But some noise is needed as people using OpenSearch in production don't seem to care we are not being able to run the test suite, so probably promising some 'OpenSearch support' that we cannot guarantee in any way. And if there's a problem with OpenSearch, we (devs) cannot launch a working environment to fix/test.

My two cents on this matter.
Comment 1 Tomás Cohen Arazi (tcohen) 2024-11-04 19:21:43 UTC
This is the last Jenkins run before it got borked and never worked again:

https://jenkins.koha-community.org/job/Koha_Main_OS2/38
Comment 2 David Cook 2024-11-04 22:32:49 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #0)
> We are not going to remove it, I guess. But some noise is needed as people
> using OpenSearch in production don't seem to care we are not being able to
> run the test suite, so probably promising some 'OpenSearch support' that we
> cannot guarantee in any way. And if there's a problem with OpenSearch, we
> (devs) cannot launch a working environment to fix/test.

As someone using OpenSearch in production, I wouldn't say that I don't care. I totally forgot that this was an issue, so thanks for making the noise and drawing my attention. 

I thought that I provided some advice on this one previously, but I can't recall where now.

With my koha-testing-docker setup on Windows, I don't think the "ktd" command works, so I don't/can't use it.

I'll look at figuring out my own test plan.
Comment 3 David Cook 2024-11-04 22:35:47 UTC
Based off a comment on bug 33563 maybe it was just something Magnus and I discussed privately for a prod install...
Comment 4 David Cook 2024-11-04 22:55:14 UTC Comment hidden (obsolete)
Comment 5 David Cook 2024-11-04 23:17:35 UTC Comment hidden (obsolete)
Comment 6 David Cook 2024-11-05 01:18:13 UTC
As I note on the koha-testing-docker Gitlab issue, it appears that version 8 of the Search::Elasticsearch Perl library arbitrarily prevents connections to Opensearch (and any other Elasticsearch-based product that doesn't provide a particular HTTP header in its HTTP response).

Technically speaking, due to the lack of encapsulation in Perl code, it's trivial to turn off this product checking from Koha code.

I suppose the question is whether or not people want to workaround this feature of the Search::Elasticsearch Perl library.

As I note on the koha-testing-docker Gitlab issue, we can get the Opensearch integration working again, but do we want to?
Comment 7 David Cook 2024-11-05 01:19:25 UTC
This is how you'd do it if you wanted to:

diff --git a/Koha/SearchEngine/Elasticsearch.pm b/Koha/SearchEngine/Elasticsearch.pm
index 5ef5235720..1cee774d41 100644
--- a/Koha/SearchEngine/Elasticsearch.pm
+++ b/Koha/SearchEngine/Elasticsearch.pm
@@ -38,6 +38,8 @@ use Clone qw( clone );
 use Modern::Perl;
 use Readonly qw( Readonly );
 use Search::Elasticsearch;
+use Search::Elasticsearch::Role::Cxn;
+$Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = '';
 use Try::Tiny qw( catch try );
 use YAML::XS;
Comment 8 David Cook 2024-11-05 01:20:58 UTC
Keeping in mind that in the future the variable scoping of that variable could change and even the looseness of Perl wouldn't prevent this workaround from failing.
Comment 9 David Cook 2024-11-05 01:23:00 UTC
Of course, you could put a proxy in front of your Opensearch and inject the HTTP header.

Or you could clone Search::Elasticsearch::Cxn::HTTPTiny, write a hack into a new version, and then update the config in koha-conf.xml to point to that new Cxn library.

Lots of possible workarounds, but it would be going against the intention of the library.

I'll leave it up to you folk to determine what you'd like to do.
Comment 10 Tomás Cohen Arazi (tcohen) 2024-11-20 03:53:36 UTC
(In reply to David Cook from comment #7)
> This is how you'd do it if you wanted to:
> 
> diff --git a/Koha/SearchEngine/Elasticsearch.pm
> b/Koha/SearchEngine/Elasticsearch.pm
> index 5ef5235720..1cee774d41 100644
> --- a/Koha/SearchEngine/Elasticsearch.pm
> +++ b/Koha/SearchEngine/Elasticsearch.pm
> @@ -38,6 +38,8 @@ use Clone qw( clone );
>  use Modern::Perl;
>  use Readonly qw( Readonly );
>  use Search::Elasticsearch;
> +use Search::Elasticsearch::Role::Cxn;
> +$Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = '';
>  use Try::Tiny qw( catch try );
>  use YAML::XS;

Let's go with this. That said, I launched an OpenSearch instance on DigitalOcean and didn't manage to make this work.
Comment 11 David Cook 2024-11-20 22:47:43 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #10)
> (In reply to David Cook from comment #7)
> > +use Search::Elasticsearch::Role::Cxn;
> > +$Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = '';
> >  use Try::Tiny qw( catch try );
> >  use YAML::XS;
> 
> Let's go with this. That said, I launched an OpenSearch instance on
> DigitalOcean and didn't manage to make this work.

Were you able to get it working with koha-testing-docker?

Sometime in the next year, I'll probably be doing a series of upgrades, which will expose me to the problem. That might not be helpful, but I'd have more real-world info then...
Comment 12 David Cook 2024-11-25 23:44:37 UTC
Just looking at my history... and it looks like I used "koha/elasticsearch-icu:opensearch-2.x" for my testing.
Comment 13 David Cook 2024-11-25 23:48:12 UTC
If you couldn't get it to work, you might also need to add the following:

$Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_HEADER= 'NONEXISTENT_HEADER';

--

If we patch this like I described before, it would disable the product check for all ES/OS. Is that what we'd want?
Comment 14 Jake Deery 2024-11-26 11:24:05 UTC
(In reply to David Cook from comment #7)
> This is how you'd do it if you wanted to:
> 
> diff --git a/Koha/SearchEngine/Elasticsearch.pm
> b/Koha/SearchEngine/Elasticsearch.pm
> index 5ef5235720..1cee774d41 100644
> --- a/Koha/SearchEngine/Elasticsearch.pm
> +++ b/Koha/SearchEngine/Elasticsearch.pm
> @@ -38,6 +38,8 @@ use Clone qw( clone );
>  use Modern::Perl;
>  use Readonly qw( Readonly );
>  use Search::Elasticsearch;
> +use Search::Elasticsearch::Role::Cxn;
> +$Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = '';
>  use Try::Tiny qw( catch try );
>  use YAML::XS;

Just applied this patch and had a go, and it seemed to work for me. There are a few failing tests, but it launches and is searchable. I'll see if I can figure out why the test is failing. For reference:

> [10:04:49] t/db_dependent/Search.t ........................................
> Dubious, test returned 255 (wstat 65280, 0xff00)
> Failed 3/3 subtests
Comment 15 David Cook 2024-11-26 22:49:41 UTC
(In reply to Jake Deery from comment #14)
> Just applied this patch and had a go, and it seemed to work for me. There
> are a few failing tests, but it launches and is searchable. I'll see if I
> can figure out why the test is failing. For reference:
> 
> > [10:04:49] t/db_dependent/Search.t ........................................
> > Dubious, test returned 255 (wstat 65280, 0xff00)
> > Failed 3/3 subtests

I'm taking a look at t/db_dependent/Search.t and I don't think it actually connects to Elasticsearch/OS, so that's interesting.

What are the tests that are failing?
Comment 16 Jonathan Druart 2025-01-30 08:40:16 UTC
any updates here?
Comment 17 David Cook 2025-01-30 23:04:20 UTC
(In reply to Jonathan Druart from comment #16)
> any updates here?

I'm still using OpenSearch in prod, so if anyone has any questions/concerns I'm happy to look at them. 

Do we want to use the workaround I suggested or leave it up to people to customize Koha and use the workaround there?
Comment 18 David Nind 2025-01-31 00:13:33 UTC
I'm happy to add the workaround instructions to the installation docs.

If we can fix it (and someone does the patches), I think we should continue to support it. (Particularly for those where licensing is important for their tech stack.)
Comment 19 David Cook 2025-01-31 00:52:27 UTC
(In reply to David Nind from comment #18)
> I'm happy to add the workaround instructions to the installation docs.
> 
> If we can fix it (and someone does the patches), I think we should continue
> to support it. (Particularly for those where licensing is important for
> their tech stack.)

Well the workaround is to the code so that makes it trickier. They'd have to maintain their own local version to avoid it getting splatted with regular updates.
Comment 20 Katrin Fischer 2025-01-31 08:21:21 UTC
It also won't fix the 2 failing Jenkins nodes. So I guess the question is if the workaround is something we could ship or not?
Comment 21 Tomás Cohen Arazi (tcohen) 2025-02-07 20:25:55 UTC
(In reply to David Cook from comment #19)
> (In reply to David Nind from comment #18)
> > I'm happy to add the workaround instructions to the installation docs.
> > 
> > If we can fix it (and someone does the patches), I think we should continue
> > to support it. (Particularly for those where licensing is important for
> > their tech stack.)
> 
> Well the workaround is to the code so that makes it trickier. They'd have to
> maintain their own local version to avoid it getting splatted with regular
> updates.

I'd say: go ahead and propose to ship our patched version of the library. Talk to Mason about dependencies tweaks in the debian control files.

This is all non sense IMHO. They went roge and disabled the competition usage. Just ship a fixed version.
Comment 22 Victor Grousset/tuxayo 2025-02-20 10:20:38 UTC
> I'd say: go ahead and propose to ship our patched version of the library.

I though the only proposed patch here and on the KTD ticket was on Koha/SearchEngine/Elasticsearch.pm ?
And not on Search::Elasticsearch itself.
Comment 23 Victor Grousset/tuxayo 2025-02-20 10:21:37 UTC
Even if forking Search::Elasticsearch wasn't really yet on the table for OpenSearch support, it might be needed sooner than we think for all future Elasticsearch support.
See Bug 39174: Search::Elasticsearch is being retired. And if ES 9 (coming likely in summer or September) doesn't work out of the box then that will be it. No support.
And if we don't find a way to make ES 9 work before 26.11 LTS release, it will end up stuck with only supporting ES 8 which will very likely be EOL before 2030-05

-------

There is an OpenSearch perl client: https://metacpan.org/dist/OpenSearch
Looking at the multiple release changelogs, I guess it should do the job for base features, but it certainly doesn't support Elasticsearch (it's not a fork of Search::Elasticsearch)
But there still might be a path there in helping to add ES support. Since there is no ES perl client anymore, if we are lucky, it might be welcomed there to have contributions from people using ES with Perl. And we are not the only ones interested in the future of ES Perl support: https://github.com/elastic/elasticsearch-perl/issues/220

So that's another option. In addition with forking Search::Elasticsearch and trying to add future OpenSearch 3.X support. And maybe proposing to the people making the OpenSearch perl lib to join forces here.

Devs at ES seem open to giving some pointers to help supporting future ES versions with perl:
https://github.com/elastic/elasticsearch-perl/issues/220#issuecomment-1196582337
Which would still help even if going the path of adding ES support to the OpenSearch perl lib.

Also at Opensearch there might be some support: https://opensearch.org/blog/community-clients/
It lists elasticsearch-perl as needing a fork. Which would get help.

Any of those paths might benefit from a name not tied to one of the products to make more clear to the users that the support is dual. And to have more chances getting some help when need from the two products. Since the user base is small, it might be tough enough already.
(Just seeding the idea for later.)
Comment 24 Victor Grousset/tuxayo 2025-02-20 10:30:47 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #21)
> This is all non sense IMHO. They went rogue and disabled the competition
> usage.

Even though ES went back to using a libre/open source license. There was lost confidence in the whole journey.
I didn't know during that time they also did this. That helps even less to regain confidence...
So one more reason part of people might feel that AWS's OpenSearch it still a safer option. And that if reasonably possible, there is still value in having it working in Koha :/ (news about OS: even if it's now under the umbrella of the Linux Foundation, still ~90% of top 30 contributors last year are AWS employees)

Both parties made somewhat logical decisions/bets to maximize their profits before anything else and in the end we pay a price in totally useless work 🤬. Especially in a language without first class library support. (having to support two libs would have been quite the useless work also)


> They went rogue and disabled the competition usage.

I quickly looked at the main official OpenSearch clients and it seems they don't even support Elasticsearch so everyone everywhere using ES/OS is screwed in some way for dual support.
Back to the above paragraph...
Comment 25 David Cook 2025-02-20 23:18:33 UTC
@Victor: That's good to know!

Worst case scenario: We write our own client. The handy thing about OS/ES is that it's a HTTP API. Doing searches against the HTTP API is super easy.

If we struggled with the indexing side of things, we could always implement the background worker in another language with better support. 

But it sucks that we're in this position :/.
Comment 26 David Cook 2025-09-12 02:54:00 UTC
Okie dokes... trying out Koha 24.11 with OpenSearch 2.

If you edit "/usr/share/koha/lib/Koha/SearchEngine/Elasticsearch.pm" to have:
use Search::Elasticsearch::Role::Cxn;
$Search::Elasticsearch::Role::Cxn::PRODUCT_CHECK_VALUE = '';

That should be the only Koha code change that would be needed.

However, if you're using HTTPS and self-signed SSL certs, there's more problems.

Search::Elasticsearch uses HTTP::Tiny for its requests, and in version 0.083 of that library it changed the default of "verify_SSL" from false to true. 

To bypass this, you can add <cxn>LWP</cxn> to your <elasticsearch> block in koha-conf.xml

Alternatively, if you want to stay with HTTP::Tiny, you can use this config in your koha-conf.xml:

<handle_args><verify_SSL>0</verify_SSL></handle_args>

I don't love the way Search::Elasticsearch::Cxn::HTTPTiny is coded. Took a bit of work to figure out that config option but oh well.
Comment 27 David Cook 2025-09-12 02:55:17 UTC
(In reply to David Cook from comment #26)
> However, if you're using HTTPS and self-signed SSL certs, there's more
> problems.

Actually, this should be true for everyone using the Search::Elasticsearch library with any flavour of ES.
Comment 28 Michał 2025-11-08 00:37:06 UTC
(In reply to David Cook from comment #26)
> <handle_args><verify_SSL>0</verify_SSL></handle_args>

(In reply to David Cook from comment #27)
> (In reply to David Cook from comment #26)
> > However, if you're using HTTPS and self-signed SSL certs, there's more
> > problems.
> 
> Actually, this should be true for everyone using the Search::Elasticsearch
> library with any flavour of ES.

OMG, yes, it is true. Thanks for sharing this, I probably wouldn't have figured it out otherwise... Added it to the wiki under default setup instructions. I was definitely trolled by that after updating to Debian 13.

> I don't love the way Search::Elasticsearch::Cxn::HTTPTiny is coded. Took a bit of work to figure out that config option but oh well.

...and the reason I wouldn't have figured it out is that I was assuming all the time that ::AEHTTP is used instead, because it has this `/certificate verify failed/ ? 'SSL'` that appears in verbose logs with HTTPTiny implementation as well, and the [SSL] category is also shown, so I figured yeah that's a no-brainer that's the HTTP client in use then...
Comment 29 David Cook 2025-11-10 22:22:59 UTC
(In reply to Michał from comment #28)
> OMG, yes, it is true. Thanks for sharing this, I probably wouldn't have
> figured it out otherwise... Added it to the wiki under default setup
> instructions. I was definitely trolled by that after updating to Debian 13.

I'm glad that my comments helped someone! It was not fun figuring it out haha.
Comment 30 Jake Deery 2025-11-12 09:22:13 UTC
Circling back to OS support and Koha, do we think we are close to deciding a path forward? It would be great to move this a step forward, for future certainty, if nothing else.
Comment 31 Tomás Cohen Arazi (tcohen) 2025-11-12 13:06:39 UTC
(In reply to Jake Deery from comment #30)
> Circling back to OS support and Koha, do we think we are close to deciding a
> path forward? It would be great to move this a step forward, for future
> certainty, if nothing else.

If there are no caveats on redistributing the library (legal concerns, etc) I'd ship our own copy of the patched library to restore support.
Comment 32 David Nind 2025-11-12 17:52:23 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #31)
> (In reply to Jake Deery from comment #30)
> > Circling back to OS support and Koha, do we think we are close to deciding a
> > path forward? It would be great to move this a step forward, for future
> > certainty, if nothing else.
> 
> If there are no caveats on redistributing the library (legal concerns, etc)
> I'd ship our own copy of the patched library to restore support.

+1
Comment 33 David Cook 2025-11-12 23:07:06 UTC
(In reply to Tomás Cohen Arazi (tcohen) from comment #31)
> (In reply to Jake Deery from comment #30)
> > Circling back to OS support and Koha, do we think we are close to deciding a
> > path forward? It would be great to move this a step forward, for future
> > certainty, if nothing else.
> 
> If there are no caveats on redistributing the library (legal concerns, etc)
> I'd ship our own copy of the patched library to restore support.

It's not necessary to patch their library though? If we change some package/global variables from our code, that's sufficient.
Comment 34 David Cook 2025-11-12 23:08:29 UTC
(In reply to Jake Deery from comment #30)
> Circling back to OS support and Koha, do we think we are close to deciding a
> path forward? It would be great to move this a step forward, for future
> certainty, if nothing else.

It's a good question. I think if Koha does remove OS support, then we'll probably move off OS and back to Elasticsearch. It's not my call to make, but I think that was our position. 

For now we're using OS as I've been able to work around limitations in Search::Elasticsearch, but that's because it's been more convenient for us than switching to Elasticsearch I think.
Comment 35 Jonathan Druart 2025-12-15 14:48:22 UTC
Created attachment 190500 [details] [review]
Bug 38345: Restore support for OpenSearch

The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.

```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```

We need to workaround Elasticsearch/OpenSearch product check
incompatibility.

This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.

Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
 ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
 ktd --shell
 perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
 prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results

6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.

IF you get a 403 during the reindex you can try to add
  cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "persistent": {
      "cluster.blocks.create_index": false
    }
  }'
```
OS thinks we lack disk space and force all indices into read-only.
Comment 36 David Nind 2025-12-15 22:27:43 UTC
OS1 and OS2 now work, however I wasn't able to start up KTD with ES6, ES7, or ES8.

Testing notes (using KTD):

1. Before following the test plan:
   - Start ktd up normally
   - Apply the patch
   - Shut KTD down

2. Starting KTD up with OS1 worked as expected.

3. For OS2 I had to change the .yml file and add: cluster.routing.allocation.disk.threshold_enabled: "false" (the curl command didn't work for me).

4. I couldn't successfully start up KTD with ES7, ES7, or ES8:

Not able to start up KTD with --es6, koha-1 exits with this error:

koha-1  | Exception 'Koha::Exception' thrown 'Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Validation Failed: 1: mapping type is missing;". Index needs to be recreated and reindexed'
db-1    | 2025-12-15 22:07:19 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-15 22:07:19 15 [Warning] Aborted connection 15 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-15 22:07:19 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
koha-1 exited with code 1
es-1    | [2025-12-15T22:07:31,661][WARN ][o.e.c.r.a.DiskThresholdMonitor] [1MMzRLW] high disk watermark [90%] exceeded on [1MMzRLWLTHKJP94Itia3EQ][1MMzRLW][/usr/share/elasticsearch/data/nodes/0] free: 11.8gb[5.8%], shards will be relocated away from this node

Not able to start up KTD with --es7, koha-1 exits:

koha-1  | [1302] Checking state of biblios index
db-1    | 2025-12-15 22:09:29 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-15 22:09:29 15 [Warning] Aborted connection 15 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-15 22:09:29 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
koha-1  | [NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 434.
koha-1 exited with code 1
es-1    | {"type": "server", "timestamp": "2025-12-15T22:10:16,345Z", "level": "WARN", "component": "o.e.c.r.a.DiskThresholdMonitor", "cluster.name": "docker-cluster", "node.name": "8945ee87acb5", "message": "high disk watermark [90%] exceeded on [eiZQBEnWQY62uuLy6r5fzg][8945ee87acb5][/usr/share/elasticsearch/data/nodes/0] free: 10.9gb[5.3%], shards will be relocated away from this node; currently relocating away shards totalling [0] bytes; the node is expected to continue to exceed the high disk watermark when these relocations are complete", "cluster.uuid": "cgLqqwuxSiaN2lOScLuGiQ", "node.id": "eiZQBEnWQY62uuLy6r5fzg"  }

Not able to start up KTD with --es8, koha-1 exits (although I haven't been able to start KTD with ES8 for some time):

koha-1  | [1302] Checking state of biblios index
db-1    | 2025-12-15 22:20:45 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-15 22:20:45 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-15 22:20:45 15 [Warning] Aborted connection 15 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
koha-1  | [NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 434.
koha-1 exited with code 1
es-1    | {"@timestamp":"2025-12-15T22:21:03.929Z", "log.level": "WARN", "message":"flood stage disk watermark [95%] exceeded on [6UmsyX5eQHWBfthqBynlDg][6c11e0699197][/usr/share/elasticsearch/data] free: 9.4gb[4.6%], all indices on this node will be marked read-only", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[6c11e0699197][management][T#3]","log.logger":"org.elasticsearch.cluster.routing.allocation.DiskThresholdMonitor","elasticsearch.cluster.uuid":"TuxajZmXQIqnbjYIYi9-YQ","elasticsearch.node.id":"6UmsyX5eQHWBfthqBynlDg","elasticsearch.node.name":"6c11e0699197","elasticsearch.cluster.name":"docker-cluster"}
Comment 37 Jonathan Druart 2025-12-16 08:51:18 UTC
Created attachment 190531 [details] [review]
Bug 38345: Restore support for OpenSearch

The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.

```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```

We need to workaround Elasticsearch/OpenSearch product check
incompatibility.

This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.

Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
 ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
 ktd --shell
 perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
 prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results

6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.

IF you get a 403 during the reindex you can try to add
  cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "persistent": {
      "cluster.blocks.create_index": false
    }
  }'
```
OS thinks we lack disk space and force all indices into read-only.
Comment 38 Jonathan Druart 2025-12-16 08:52:06 UTC
(In reply to David Nind from comment #36)
> 4. I couldn't successfully start up KTD with ES7, ES7, or ES8:

Oops I am sorry, last minute change broke everything!
The condition was actually commented...
Comment 39 David Nind 2025-12-16 10:31:36 UTC
Can now get to work with OS1, OS2 and ES7.

Tests passed for ES7, but didn't for OS1 and OS2:

Test Summary Report
-------------------
t/db_dependent/Koha/SearchEngine/Elasticsearch/Indexer.t     (Wstat: 256 (exited 1) Tests: 5 Failed: 1)
  Failed test:  5
  Non-zero exit status: 1
t/db_dependent/Koha/SearchEngine/Elasticsearch/Search.t      (Wstat: 256 (exited 1) Tests: 15 Failed: 1)
  Failed test:  15
  Non-zero exit status: 1
Files=5, Tests=58, 13 wallclock secs ( 0.03 usr  0.01 sys +  6.95 cusr  1.87 csys =  8.86 CPU)
Result: FAIL


On ES8, koha-1 exited with code 1 (but haven't managed to get ES8 running for some time on KTD):

koha-1  | Use of uninitialized value in string ne at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch.pm line 52.
koha-1  | [Timeout] ** [http://es:9200]-[599] Timed out while waiting for socket to become ready for reading, called from sub Search::Elasticsearch::Role::Cxn::process_response at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch.pm line 55. With vars: {'status_code' => 599,'request' => {'serialize' => 'std','body' => {'settings' => {'index.number_of_shards' => 5,'index' => {'analysis' => {'char_filter' => {'facet' => {'pattern' => '\\s*(?<!\\p{Lu})[.\\-,;]*\\s*$','type' => 'pattern_replace','replacement' => ''},'punctuation' => {'type' => 'pattern_replace','replacement' => '','pattern' => '([\\x00-\\x1F,\\x21-\\x2F,\\x3A-\\x40,\\x5B-\\x60,\\x7B-\\x89,\\x8B,\\x8D,\\x8F,\\x90-\\x99,\\x9B,\\x9D,\\xA0-\\xBF,\\xD7,\\xF7])'}},'analyzer' => {'analyzer_standard' => {'tokenizer' => 'icu_tokenizer','filter' => ['icu_folding']},'analyzer_phrase' => {'char_filter' => ['punctuation'],'tokenizer' => 'keyword','filter' => ['icu_folding']},'analyzer_stdno' => {'filter' => ['icu_folding'],'tokenizer' => 'whitespace','char_filter' => ['punctuation']}},'normalizer' => {'nfkc_cf_normalizer' => {'char_filter' => 'icu_normalizer','type' => 'custom'},'icu_folding_normalizer' => {'type' => 'custom','filter' => ['icu_folding']},'facet_normalizer' => {'char_filter' => 'facet'}}}},'index.mapping.total_fields.limit' => 10000,'index.number_of_replicas' => 1,'index.max_result_window' => 1000000}},'ignore' => [],'method' => 'PUT','path' => '/koha_kohadev_biblios','qs' => {},'mime_type' => 'application/json'}}
db-1    | 2025-12-16 10:16:53 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-16 10:16:53 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-16 10:16:53 15 [Warning] Aborted connection 15 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
koha-1 exited with code 1

ON ES6 koha-1 exited with code 1 (I can't get ES6 to start normally with KTD before this):

koha-1  | [DEPRECATION] [types removal] The parameter include_type_name should be explicitly specified in put mapping requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', and requests are expected to omit the type name in mapping definitions. - In request: {body => {properties => {abstract => {analyzer => "analyzer_standard",fields => {ci_raw => {normalizer => "icu_fold
....
koha-1  | Exception 'Koha::Exception' thrown 'Unable to update mappings for index "koha_kohadev_biblios". Reason was: "Validation Failed: 1: mapping type is missing;". Index needs to be recreated and reindexed'
db-1    | 2025-12-16 10:24:47 13 [Warning] Aborted connection 13 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-16 10:24:47 14 [Warning] Aborted connection 14 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
db-1    | 2025-12-16 10:24:47 15 [Warning] Aborted connection 15 to db: 'koha_kohadev' user: 'koha_kohadev' host: '172.18.0.5' (Got an error reading communication packets)
koha-1 exited with code 1
Comment 40 Jonathan Druart 2025-12-16 11:32:35 UTC
(In reply to David Nind from comment #36)
> OS1 and OS2 now work, however I wasn't able to start up KTD with ES6, ES7,
> or ES8.

I have changes in my ktd's docker-compose.os2.yml file

diff --git a/docker-compose.os2.yml b/docker-compose.os2.yml
index 66a748e..bc28aea 100644
--- a/docker-compose.os2.yml
+++ b/docker-compose.os2.yml
@@ -8,6 +8,9 @@ services:
             plugins.security.disabled: "true"
             ES_JAVA_OPTS: -Xms512m -Xmx512m
             OPENSEARCH_INITIAL_ADMIN_PASSWORD: ${OPENSEARCH_INITIAL_ADMIN_PASSWORD:-pu1kohphei4heeY4pai7ohp6vei4Ea6i}
+            cluster.routing.allocation.disk.threshold_enabled: "false"
+            network.host: 0.0.0.0
+            network.publish_host: es
         ulimits:
             memlock:
                 soft: -1

I am trying now.
Comment 41 Jonathan Druart 2025-12-16 11:38:48 UTC
Created attachment 190534 [details] [review]
Bug 38345: Remove warnings

Patch from commit 484ca2d
Comment 42 Jonathan Druart 2025-12-16 11:49:21 UTC
With the changes from comment 40 and the second patch, all good for OS2 and ES7.

If those 2 pass we can already move forward I think.
Comment 43 David Nind 2025-12-16 18:11:35 UTC
Created attachment 190556 [details] [review]
Bug 38345: Restore support for OpenSearch

The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.

```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```

We need to workaround Elasticsearch/OpenSearch product check
incompatibility.

This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.

Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
 ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
 ktd --shell
 perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
 prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results

6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.

IF you get a 403 during the reindex you can try to add
  cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "persistent": {
      "cluster.blocks.create_index": false
    }
  }'
```
OS thinks we lack disk space and force all indices into read-only.

Signed-off-by: David Nind <david@davidnind.com>
Comment 44 David Nind 2025-12-16 18:11:39 UTC
Created attachment 190557 [details] [review]
Bug 38345: Remove warnings

Signed-off-by: David Nind <david@davidnind.com>
Comment 45 David Nind 2025-12-16 18:13:15 UTC
Now working for me for OS1, OS2, and ES7, and tests pass.
Comment 46 Martin Renvoize (ashimema) 2025-12-17 13:57:07 UTC
Created attachment 190580 [details] [review]
Bug 38345: Restore support for OpenSearch

The Koha_Main_OS1 and Koha_Main_OS2 jobs are currently failing on
Jenkins.

```
[NoNodes] ** No nodes are available: [http://es:9200], called from sub Search::Elasticsearch::Role::Client::Direct::__ANON__ at /kohadevbox/koha/Koha/SearchEngine/Elasticsearch/Indexer.pm line 417. # Looks like your test exited with 11 just after 3.
```

We need to workaround Elasticsearch/OpenSearch product check
incompatibility.

This patch suggests to monkey-patch
Search::Elasticsearch::Role::Cxn::process_response to disable the strict
product header check at runtime. This allows the same codebase to talk
to both Elasticsearch 8+ and OpenSearch without forking or patching
upstream.

Test plan:
0. Set KOHA_ELASTICSEARCH=yes in ktd's .env
1. Startup ktd with the compose file for OS1
 ktd -f docker-compose.os1.yml up
2. Set SearchEngine=Elasticsearch
3. reindex
 ktd --shell
 perl misc/search_tools/rebuild_elasticsearch.pl -a -b
4. run the tests
 prove -r t/db_dependent/Koha/SearchEngine/Elasticsearch
5. search!
http://localhost:8081/cgi-bin/koha/catalogue/search.pl?q=perl
should show results

6. ktd down then repeat using the compose file for OS2 (docker-compose.os2.yml), ES6 and ES7
for ES6 and ES7 you can use the --es6 and --es7 ktd flag.

IF you get a 403 during the reindex you can try to add
  cluster.routing.allocation.disk.threshold_enabled: "false"
to the environment section of the compose file.
Or
```
curl -X PUT http://es:9200/_cluster/settings \
  -H 'Content-Type: application/json' \
  -d '{
    "persistent": {
      "cluster.blocks.create_index": false
    }
  }'
```
OS thinks we lack disk space and force all indices into read-only.

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 47 Martin Renvoize (ashimema) 2025-12-17 13:57:09 UTC
Created attachment 190581 [details] [review]
Bug 38345: Remove warnings

Signed-off-by: David Nind <david@davidnind.com>
Signed-off-by: Martin Renvoize <martin.renvoize@openfifth.co.uk>
Comment 48 Martin Renvoize (ashimema) 2025-12-17 13:57:27 UTC
Nice one!
Comment 49 Lucas Gass (lukeg) 2025-12-17 20:26:09 UTC
Nice work everyone!

Pushed to main for 26.05
Comment 50 Jacob O'Mara 2026-01-16 16:17:47 UTC
Thanks all, pushed to 25.11.x