Bug 39314

Summary: RabbitMQ can't upgrade from 3.x to 4.x without running a data migration on 3.13
Product: Koha Reporter: Victor Grousset/tuxayo <victor>
Component: Installation and upgrade (command-line installer)Assignee: Bugs List <koha-bugs>
Status: In Discussion --- QA Contact: Testopia <testopia>
Severity: major    
Priority: P5 - low CC: aleisha, davewood, dcook, jonathan.druart, oliver.behnke, tomascohen
Version: Main   
Hardware: All   
OS: All   
See Also: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=41506
GIT URL: Initiative type: ---
Sponsorship status: --- Comma delimited list of Sponsors:
Crowdfunding goal: 0 Patch complexity: ---
Documentation contact: Documentation submission:
Text to go in the release notes:
Version(s) released in:
Circulation function:

Description Victor Grousset/tuxayo 2025-03-12 08:45:05 UTC
Reported to Debian as it affects their next release which will do a jump from 3.10 to 4.0: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#10

Which can't work: https://www.rabbitmq.com/docs/upgrade
«You can only upgrade to RabbitMQ 4.0 from RabbitMQ 3.13.» 

«stable feature flags have to be enabled before the upgrade. The upgrade 
will fail if you miss this step.»

See there more more details.

-----

For us: is there still data in RabbitMQ when stopping a Koha instance and upgrading it? I recall reading that the DB was basically mirroring the message queue data during a discussion about the actual need for a dedicated message queue. Does it mean the koha-common package can wipe the RabbitMQ data which will allow it to start without issue?
Comment 1 David Cook 2025-03-14 01:57:36 UTC
(In reply to Victor Grousset/tuxayo from comment #0)
> Reported to Debian as it affects their next release which will do a jump
> from 3.10 to 4.0:
> https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#10
> 
> Which can't work: https://www.rabbitmq.com/docs/upgrade
> «You can only upgrade to RabbitMQ 4.0 from RabbitMQ 3.13.» 

That's fun...

> For us: is there still data in RabbitMQ when stopping a Koha instance and
> upgrading it? I recall reading that the DB was basically mirroring the
> message queue data during a discussion about the actual need for a dedicated
> message queue. Does it mean the koha-common package can wipe the RabbitMQ
> data which will allow it to start without issue?

The DB stores the jobs, and the Koha app passes messages to background workers via RabbitMQ.

If the background worker starts up and can't get a connection to RabbitMQ, it'll query the DB directly for the jobs. 

What this means is that if you stopped koha-common, nuked RabbitMQ, re-installed RabbitMQ, and started koha-common, you could have data loss if there were any messages in RabbitMQ that hadn't been processed.
Comment 2 David Cook 2025-03-14 02:00:52 UTC
A dedicated sysadmin who knows to look out for the RabbitMQ issue could perform a RabbitMQ reinstall without a problem.

But for someone without sysadmin support... there could be problems depending when the upgrade is performed.

=-

I'm trying to think of workarounds that tried to determine if there was any pending messages and re-queue or otherwise deal with those messages... but it would probably be messy.
Comment 3 Victor Grousset/tuxayo 2025-03-14 07:59:58 UTC
> A dedicated sysadmin who knows to look out for the RabbitMQ issue could perform a RabbitMQ reinstall without a problem

They would have to find a package somewhere for the latest 3.X just for the upgrade. Still quite the hassle >_<
Comment 4 David Cook 2025-03-16 23:47:48 UTC
(In reply to Victor Grousset/tuxayo from comment #3)
> > A dedicated sysadmin who knows to look out for the RabbitMQ issue could perform a RabbitMQ reinstall without a problem
> 
> They would have to find a package somewhere for the latest 3.X just for the
> upgrade. Still quite the hassle >_<

Sure, but it's not uncommon. I think Keycloak requires stepped upgrades for certain versions. PostgreSQL too for really old versions. 

We try to make things seamless for people, but I think there needs to be limits too.
Comment 5 Jonathan Druart 2025-05-12 07:35:34 UTC
*** Bug 39868 has been marked as a duplicate of this bug. ***
Comment 6 Jonathan Druart 2025-05-12 07:35:51 UTC
There is also https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104149
Comment 7 Jonathan Druart 2025-11-06 09:18:39 UTC
Is this still valid?

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1104149#10
"I wasn't able to reproduce, so probably it does work now?"
Comment 8 Victor Grousset/tuxayo 2025-12-16 03:13:23 UTC
It seems these turn out to be different problems. See 39868 bug comment 4
Comment 9 Victor Grousset/tuxayo 2025-12-16 03:13:35 UTC
On the Debian side, there isn't much they can do without causing other issues.

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#44
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1100165#51

At least there are some strategies:
> Heavy RabbitMQ users that cannot reset their /var/lib/rabbitmq/mnesia 
> database could maybe use upstream docker image to attempt a rolling 
> upgrade. Otherwise, users are advised to simply drain all queues and 
> make clients point to a new cluster if that's possible.

At least the Debian release notes should warn about this:
> writing about it in the Trixie release notes

I don't know about Ubuntu.
And whether that's enough anyway.

Should we put a note on the release notes of Bug 41077?
Wait, that wouldn't work since there is no patch. Same if we put something in the release notes here.
Any ideas?

(In reply to David Cook from comment #1)
> tuxayo:
> > For us: is there still data in RabbitMQ when stopping a Koha instance and
> > upgrading it? [...]
> 
> [...]
> What this means is that if you stopped koha-common, nuked RabbitMQ,
> re-installed RabbitMQ, and started koha-common, you could have data loss if
> there were any messages in RabbitMQ that hadn't been processed.

Is it something that regularly happens? (have messages in RabbitMQ that haven't been processed) With the reasonable assumption that people don't upgrade Koha in the middle of a day with librarians doing things that use the queue/workers.

Or is it something that totally happens?
Maybe the assumption is wrong or there is another source of messages that is unavoidable.
Comment 10 Oliver Behnke 2026-01-08 14:29:57 UTC
Just ran into this issue myself and resolved it (before I came here) by "simply" installing the intermediate versions required, one by one, enabling their feature flags accordingly. Pretty annoying but I honestly don't see a way around that, presuming that Debian releases will always jump accross a few minor releases of RabbitMQ.

It's all outlined in [1] and [2]. That latter also tells you which intermediate releases you need to install:
- Look at the first column ("Require") of that table and find the next version higher than your current one *and* where the second column ("Stable") has a version higher than your's
- Then check the second column and find the highest version: that's your next target
- Install it, enable all features, then repeat

For example:
1) Debian bookworm provides 3.10.8 which means 3.12.0 is the next upgrade step that needs flags which your version doesn't yet include. So you want to upgrade to at least 3.11.0 in this case (the latest revision is 3.11.28) to prepare for 3.12
2) Debian trixie comes with 4.0.5 which also happens to be the next upgrade target to prepare for. According to the flags table 3.13.1 is the latest release introducing a new flag for 4.0, so that's what we install (the latest revision is 3.13.28). Note: there's no need to install 3.12 first since 3.13 did *not* have any additional flag requirements.

However, to make things worse RabbitMQ releases depend on different Erlang versions. And to make it even more worse, they require a specific major version. E.g. 3.10 depends on Erlang 25, whereas 3.13 depends on Erlang 26, while 4.0 depends on Erlang 27. Yeah, so we also get to mess with those!

Fortunately, they provide Debian repos [3] for Erlang and RabbitMQ making things a bit easier. But be sure to pin those repos if you want to stick to Debian's packages by default.

Hope this helps

[1] https://www.rabbitmq.com/blog/2022/07/20/required-feature-flags-in-rabbitmq-3.11
[2] https://www.rabbitmq.com/docs/feature-flags#core-feature-flags
[3] https://www.rabbitmq.com/docs/install-debian#apt-repositories
Comment 11 davewood 2026-02-11 10:17:40 UTC
@Oliver Behnke

I think you made a mistake with the rabbitmq versions.

There is no version 3.13.28 according to https://www.rabbitmq.com/release-information

Latest 3.13.x release is 3.13.7

ps: thanks for the infos you shared!
Comment 12 davewood 2026-02-11 13:00:21 UTC
i started fixing our broken debian13 rabbitmq-server packages and this list of commands works for us at HKS3 (www-koha-support.eu) to fix the issue.

```
curl -1sLf "https://keys.openpgp.org/vks/v1/by-fingerprint/0A9AF2115F4687BD29803A206B73A36E6026DFCA" |  gpg --dearmor |  tee /usr/share/keyrings/com.rabbitmq.team.gpg > /dev/null

tee /etc/apt/sources.list.d/rabbitmq.list <<EOF
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb1.rabbitmq.com/rabbitmq-erlang/debian/bookworm bookworm main
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb2.rabbitmq.com/rabbitmq-erlang/debian/bookworm bookworm main
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb1.rabbitmq.com/rabbitmq-server/debian/trixie trixie main
deb [arch=amd64 signed-by=/usr/share/keyrings/com.rabbitmq.team.gpg] https://deb2.rabbitmq.com/rabbitmq-server/debian/trixie trixie main
EOF

apt update

apt install rabbitmq-server=3.11.28-1 erlang-base:amd64=1:25.3.2.21-1 erlang-crypto:amd64=1:25.3.2.21-1 erlang-ssl:amd64=1:25.3.2.21-1 erlang-public-key:amd64=1:25.3.2.21-1 erlang-asn1:amd64=1:25.3.2.21-1 erlang-runtime-tools:amd64=1:25.3.2.21-1 erlang-mnesia:amd64=1:25.3.2.21-1 erlang-eldap:amd64=1:25.3.2.21-1 erlang-tools:amd64=1:25.3.2.21-1 erlang-os-mon:amd64=1:25.3.2.21-1 erlang-snmp:amd64=1:25.3.2.21-1 erlang-inets:amd64=1:25.3.2.21-1 erlang-ftp:amd64=1:25.3.2.21-1 erlang-tftp:amd64=1:25.3.2.21-1 erlang-parsetools:amd64=1:25.3.2.21-1 erlang-xmerl:amd64=1:25.3.2.21-1 erlang-syntax-tools:amd64=1:25.3.2.21-1

# optional if last step fails with {case_clause,version_not_available} error
rm -rf /var/lib/rabbitmq/mnesia/rabbit*

systemctl restart rabbitmq-server.service
rabbitmqctl enable_feature_flag all

apt install rabbitmq-server=3.13.7-1 erlang-base:amd64=1:26.2.5.14-1 erlang-xmerl:amd64=1:26.2.5.14-1 erlang-os-mon:amd64=1:26.2.5.14-1 erlang-mnesia:amd64=1:26.2.5.14-1 erlang-snmp:amd64=1:26.2.5.14-1 erlang-crypto:amd64=1:26.2.5.14-1 erlang-runtime-tools:amd64=1:26.2.5.14-1 erlang-ssl:amd64=1:26.2.5.14-1 erlang-public-key:amd64=1:26.2.5.14-1 erlang-asn1:amd64=1:26.2.5.14-1 erlang-eldap:amd64=1:26.2.5.14-1 erlang-inets:amd64=1:26.2.5.14-1 erlang-ftp:amd64=1:26.2.5.14-1 erlang-tftp:amd64=1:26.2.5.14-1 erlang-tools:amd64=1:26.2.5.14-1  erlang-syntax-tools:amd64=1:26.2.5.14-1 erlang-parsetools:amd64=1:26.2.5.14-1

systemctl restart rabbitmq-server.service
rabbitmqctl enable_feature_flag all

rm /etc/apt/sources.list.d/rabbitmq.list
rm /usr/share/keyrings/com.rabbitmq.team.gpg
apt update
apt upgrade

systemctl restart rabbitmq-server.service
rabbitmqctl enable_feature_flag all

```
Comment 13 Oliver Behnke 2026-02-11 13:10:30 UTC
(In reply to davewood from comment #11)
> I think you made a mistake with the rabbitmq versions.
> 
> There is no version 3.13.28 according to
> https://www.rabbitmq.com/release-information
> 
> Latest 3.13.x release is 3.13.7

Oops, you're right. I probably confused the patch level with 3.11.28 (above). Sorry.

> ps: thanks for the infos you shared!

Glad they helped.