|
Description
Katrin Fischer
2025-12-31 12:51:14 UTC
Adding some more personal notes: * When we had a survey at the hackfest in Marseille last year it showed that almost all of the big providers have deactivated RabbitMQ in their installations. That certainly impacts the amount of testing and development work in that area. * Currently RabbitMQ is a dependency that is installed with the Koha packages. So currently it requires some hacking to make sure it's not installed and running even when the new pref JobsNotificationMethod is set to SQL polling. I'll link open RabbitMQ related bugs in the see also section. Yes, please, even if it's only because of this: https://bugs.koha-community.org/bugzilla3/show_bug.cgi?id=39314#c10 Cheers I agree it should be removed. It was a great idea in theory, but in practice it simply doesn't work well. I'm in favor of removing the use of rabbit (I submitted such patch on bug 35920 already) and we can always rethink the architecture and reintroduce with a better split of concerns (e.g. adding a task manager/scheduler, etc). I agree with removing this. We have never seen benefit of using it since background jobs seem to work smoothly even without it. So it's main purpose has been to fill our test logs with warnings of not founding Stomp server. I'm on the fence (surprise surprise). I don't think that I feel strongly one way or another anymore, but I've got some thoughts (not quite pros/cons) below if you want to read them. -- I haven't used the SQL polling method for the background workers, so I don't know how effective/efficient it is or how well it supports multiple workers for 1 queue, but I have heard that a lot of people like it. I also have used SQL polling for background jobs in the past and it was good enough for simple use cases. Originally, I had an interest in RabbitMQ for high performance background jobs, as I was targeting background jobs that needed to be actioned extremely quickly (ie near real-time) and needed a FIFO queue that could ensure proper distribution of tasks to multiple concurrent workers. However, the use case for that was never realised in Koha, so it didn't matter in the end. Despite a few hiccups, RabbitMQ has worked pretty well for me in Koha (and on other systems). I think there have been times where people have avoided RabbitMQ rather than engaging with it for fixing bugs. But... I think that's also a sign that maybe RabbitMQ isn't a good fit for the Koha community. If an alternative solution has grown up in reaction to it, maybe we need to listen to that. There's also the issue with RabbitMQ not being very easy to upgrade. We don't have a lot of resources in the Koha community, and supporting multiple options can be difficult. We already do it with Zebra/Elasticsearch and that's hard enough. Anyway, maybe I'll leave it there for now. Basically, I think RabbitMQ provides us with a lot of power and potential to carry Koha into the future. However, I also acknowledge that a lot of people don't like it and don't want it in Koha, and maybe it's overkill for what we need in Koha at present. When making choices on technology, I think we do need to think about what we can collectively maintain as a community. It does seem like we have more human resources for maintaining the SQL polling option than the RabbitMQ option. IMHO, to make an informed decision about this issue it'd also be very interesting to get a rough idea about the number of Libraries that have switched to SQL polling instead of the default setting which uses RabbitMQ (i.e. 'STOMP'). We don't (yet) share the JobsNotificationMethod setting with Hea, but Bug 23849 aims to change that. It might also be worth correlating the number of Libraries that use/don't use RabbitMQ with other metrics such as the number of bibliographic/authority records, the number of patrons, etc. as this could provide an estimate for how much a Library's size affects the decision of whether or not to use RabbitMQ. Just my €0.02 ;-) (In reply to Andreas Roussos from comment #7) > IMHO, to make an informed decision about this issue it'd also be > very interesting to get a rough idea about the number of Libraries > that have switched to SQL polling instead of the default setting > which uses RabbitMQ (i.e. 'STOMP'). We did a survey at the last Hackfest in Marseille and all the big support providers have turned RabbitMQ off... so that gives some indication. > We don't (yet) share the JobsNotificationMethod setting with Hea, > but Bug 23849 aims to change that. > > It might also be worth correlating the number of Libraries that > use/don't use RabbitMQ with other metrics such as the number of > bibliographic/authority records, the number of patrons, etc. as > this could provide an estimate for how much a Library's size > affects the decision of whether or not to use RabbitMQ. > > Just my €0.02 ;-) I know that KIT has deactivated RabbitMQ and we too, so we can tell it's working really well without for libraries with a million records and more. (In reply to Andreas Roussos from comment #7) > It might also be worth correlating the number of Libraries that > use/don't use RabbitMQ with other metrics such as the number of > bibliographic/authority records, the number of patrons, etc. as > this could provide an estimate for how much a Library's size > affects the decision of whether or not to use RabbitMQ. Our biggest consortium, Vaski libraries, has over 2,5 million bibliographic records, around 235900 users and 74 libraries (all the libraries use same Koha installation). Overall, most of our consortiums have over 1 million bibliographic records and there are total number of 400 libraries in consortiums. I think the question is not if libraries are using it, because I am sure there are libraries using it as it installs by defeault and couldn't be "properly" turned off before the introduction of the system preference only somewhat recently. The things I worry about are: * added complexity * will to maintain and develop * dependency and upgrade issues * lack of benefit If the SQL polling does the job and it looks like most have moved this direction now, even the real big libraries, then we might want to focus on that solution. A first backwards-compatible change could be to remove rabbitmq-server from the dependencies of the koha-common Debian package. Move it to the list of recommended packages instead, which are installed by default but can also be uninstalled. That would match the JobsNotificationMethod option introduced in bug 35655 on the package level. Unfortunately, none of the other packages (e.g. koha-core, still experimental) can currently easily replace koha-common without rabbitmq-server, so the change above would be ideal, I think. Thanks I would like to see rabbitmq-server moved to the suggested packages in koha-common. I think it should be removed completly, maybe in a first step just make SQL polling the default, and move rabbitmq-server to the suggested packages in koha-common ( as suggested by David) Morituri te salutant I never had a problem with RabbitMQ but the pressure is obvious here. So lets remove it ;) We at BibLibre use SQL Pooling on all instances. But we have only one Koha (and one DB) per virtual machine so maybe this context makes it easier. I must say there was sometimes performance issues when Elastiseach didn't had its own deamon. Now with 24.11 it is fine. Plus our hardware is very powerful so background are started less than a few seconds after creation. |