Many here recommend using Kafka or RabbitMQ for real-time notifications. While these tools work well with a relatively stable, limited set of topics, they become costly and inefficient when dealing with a large number of dynamic subscribers, such as in a messaging app where users frequently come and go. In RabbitMQ, queue bindings are resource-intensive, and in Kafka, creating new subscriptions often triggers expensive rebalancing operations. I've seen a use case for a messenger app with 100k concurrent subscribers where developers used RabbitMQ and individual queues for each user. It worked at 60 CPU on Rabbit side during normal situation and during mass reconnections of users (due to some proxy reload in infra) – it took up to several minutes for users to reconnect. I suggested switching to https://github.com/centrifugal/centrifugo with Redis engine (combines PUB/SUB + Redis streams for individual queues) – and it went to 0.3 CPU on Redis side. Now the system serves about 2 million concurrent connections.
Readit NewsFZambia commented on Postgres LISTEN/NOTIFY does not scale recall.ai/blog/postgres-l... · Posted by u/davidgu
FZambia commented on Postgres LISTEN/NOTIFY does not scale recall.ai/blog/postgres-l... · Posted by u/davidgu
I've been disappointed by Nats. Core Nats is good and works well, but if you need stronger delivery guarantees you need to use Jetstream which has a lot of quirks, for instance it does not integrate well with the permission system in Core Nats. Their client SDKs are very buggy and unreliable. I've used the Python, Rust and Go ones, only the Go one worked as expected. I would recommend using rabbitmq, Kafka or redpanda instead of Nats.
Client SDKs are often a major challenge in systems like these. In my experience, building SDKs on top of asynchronous protocols is particularly tricky. It's generally much easier to make the server-side part reliable. The complexity arises because SDKs must account for a wide range of usage patterns - and you are not controlling the usage.
Asynchronous protocols frequently result in callback-based or generator-style APIs on the client side, which are hard to implement safely and intuitively. For example, consider building a real-time SDK for something like NATS. Once a message arrives, you need to invoke a user-defined callback to handle it. At that point, you're faced with a design decision: either call the callback synchronously (which risks blocking the socket reading loop), or do it asynchronously (which raises issues like backpressure handling).
Also, SDKs are often developed by different people, each with their own design philosophy and coding style, leading to inconsistency and subtle bugs.
So this isn't only about NATS. Just last week, we ran into two critical bugs in two separate Kafka SDKs at work.
FZambia commented on Postgres LISTEN/NOTIFY does not scale recall.ai/blog/postgres-l... · Posted by u/davidgu
For real-time notifications, I believe Nats (https://nats.io) or Centrifugo (https://centrifugal.dev) are worth checking out these days. Messages may be delivered to those systems from PostgreSQL over replication protocol through Kafka as an intermediary buffer. Reliable real-time messaging comes with lots of complexities though, like late message delivery, duplicate message delivery. If the system can be built around at most once guarantees – can help to simplify the design dramatically. Depends on the use case of course, often both at least once and at most once should co-exist in one app.
Hi everyone!
I'd like to share that we've just released Centrifugo v6 - a major update of scalable WebSocket server. The release addresses some usability pain points and adds nice features and more observability.
Centrifugo is an open-source standalone server written in Go – https://github.com/centrifugal/centrifugo. Centrifugo can instantly deliver messages to application online users connected over supported transports (WebSocket, HTTP-streaming, Server-Sent Events (EventSource), GRPC, WebTransport). Centrifugo has the concept of a channel – so it's a user-facing PUB/SUB server. Everything implemented in a language-agnostic way – so Centrifugo can be used in combination with any frontend or backend stack.
These days we also provide Centrifugo PRO version – and trying to find a balance to be sustainable.
The server is based on the open-source Centrifuge library - https://github.com/centrifugal/centrifuge, so many improvements mentioned in Centrifugo v6 release blog post (even those for Centrifugo PRO) may be used just as a library in Go application.
We provide real-time SDKs for popular client environments – for browser and mobile development – they connect to both Centrifuge library based servers and Centrifugo server.
Generally Centrifugal ecosystem provides a good alternative to Socket.IO and cloud services like Pusher.com and Ably.com
Will be happy to answer on any questions
FZambia commented on Server-Sent Events (SSE) Are Underrated igorstechnoclub.com/serve... · Posted by u/Igor_Wiwi
The part where it says:
> SSE works seamlessly with existing HTTP infrastructure:
I'd be careful with that assumption. I have tried using SSE through some 3rd party load balancer at my work and it doesn't work that well. Because SSE is long-lived and doesn't normally close immediately, this load balancer will keep collecting and collecting bytes from the server and not forward it until server closes the connection, effectively making SSEs useless. I had to use WebSockets instead to get around this limitation with the load balancer.
Yep, and in addition to that the ephemeral ports problem will araise at some scale with long-lived connections and infrastructure balancer/reverse proxy chain. So it's still required to tune.
FZambia commented on Server-Sent Events (SSE) Are Underrated igorstechnoclub.com/serve... · Posted by u/Igor_Wiwi
Cool didn’t know this. I used a similar solution called Centrifugo for a while. It allows you to choose which transport to use (ws, sse, others)
Wow, it's fascinating how a single HN comment can drive meaningful traffic to a project! I'm the author of Centrifugo, and I appreciate you mentioning it here.
Let me share a bit more about Centrifugo transport choices. It’s not just about supporting multiple transports — developers can also choose between bidirectional and unidirectional communication models, depending on their needs.
For scenarios where stable subscriptions are required without sending data from the client to the server, Centrifugo seamlessly supports unidirectional transports like SSE, HTTP-streaming, unidirectional gRPC streams, and even unidirectional WebSockets (this may sound kinda funny for many I guess). This means integration is possible without relying on client-side SDKs.
However, Centrifugo truly shines in its bidirectional communication capabilities. Its primary transport is WebSocket – with JSON or Protobuf protocols, with SSE/HTTP-streaming fallbacks that are also bidirectional — an approach reminiscent of SockJS, but with more efficient implementation and no mandatory sticky sessions. Sticky sessions is an optimization in Centrifugo, not a requirement. It's worth noting that SSE only supports JSON format, since binary is not possible with it. This is where HTTP-streaming in conjuction with ReadableStream browser API can make much more sense!
I believe Centrifugo gives developers the flexibility to choose the transport and communication style that best fits their application's needs. And it scales good out of the box to many nodes – with the help of Redis or Nats brokers. Of course this all comes with limitations every abstraction brings.