FZambia

Thanks! Our users mostly run Redis or Valkey, so we optimize for them. We also have Dragonflydb in our test suite. But good to know there is one more option which may be used as a drop in replacement for Redis and provide extra properties, guarantees or performance benefits.

Many here recommend using Kafka or RabbitMQ for real-time notifications. While these tools work well with a relatively stable, limited set of topics, they become costly and inefficient when dealing with a large number of dynamic subscribers, such as in a messaging app where users frequently come and go. In RabbitMQ, queue bindings are resource-intensive, and in Kafka, creating new subscriptions often triggers expensive rebalancing operations. I've seen a use case for a messenger app with 100k concurrent subscribers where developers used RabbitMQ and individual queues for each user. It worked at 60 CPU on Rabbit side during normal situation and during mass reconnections of users (due to some proxy reload in infra) – it took up to several minutes for users to reconnect. I suggested switching to https://github.com/centrifugal/centrifugo with Redis engine (combines PUB/SUB + Redis streams for individual queues) – and it went to 0.3 CPU on Redis side. Now the system serves about 2 million concurrent connections.

Client SDKs are often a major challenge in systems like these. In my experience, building SDKs on top of asynchronous protocols is particularly tricky. It's generally much easier to make the server-side part reliable. The complexity arises because SDKs must account for a wide range of usage patterns - and you are not controlling the usage.

Asynchronous protocols frequently result in callback-based or generator-style APIs on the client side, which are hard to implement safely and intuitively. For example, consider building a real-time SDK for something like NATS. Once a message arrives, you need to invoke a user-defined callback to handle it. At that point, you're faced with a design decision: either call the callback synchronously (which risks blocking the socket reading loop), or do it asynchronously (which raises issues like backpressure handling).

Also, SDKs are often developed by different people, each with their own design philosophy and coding style, leading to inconsistency and subtle bugs.

So this isn't only about NATS. Just last week, we ran into two critical bugs in two separate Kafka SDKs at work.

For real-time notifications, I believe Nats (https://nats.io) or Centrifugo (https://centrifugal.dev) are worth checking out these days. Messages may be delivered to those systems from PostgreSQL over replication protocol through Kafka as an intermediary buffer. Reliable real-time messaging comes with lots of complexities though, like late message delivery, duplicate message delivery. If the system can be built around at most once guarantees – can help to simplify the design dramatically. Depends on the use case of course, often both at least once and at most once should co-exist in one app.

Hi everyone!

I'd like to share that we've just released Centrifugo v6 - a major update of scalable WebSocket server. The release addresses some usability pain points and adds nice features and more observability.

Centrifugo is an open-source standalone server written in Go – https://github.com/centrifugal/centrifugo. Centrifugo can instantly deliver messages to application online users connected over supported transports (WebSocket, HTTP-streaming, Server-Sent Events (EventSource), GRPC, WebTransport). Centrifugo has the concept of a channel – so it's a user-facing PUB/SUB server. Everything implemented in a language-agnostic way – so Centrifugo can be used in combination with any frontend or backend stack.

These days we also provide Centrifugo PRO version – and trying to find a balance to be sustainable.

The server is based on the open-source Centrifuge library - https://github.com/centrifugal/centrifuge, so many improvements mentioned in Centrifugo v6 release blog post (even those for Centrifugo PRO) may be used just as a library in Go application.

We provide real-time SDKs for popular client environments – for browser and mobile development – they connect to both Centrifuge library based servers and Centrifugo server.

Generally Centrifugal ecosystem provides a good alternative to Socket.IO and cloud services like Pusher.com and Ably.com

Will be happy to answer on any questions

Yep, and in addition to that the ephemeral ports problem will araise at some scale with long-lived connections and infrastructure balancer/reverse proxy chain. So it's still required to tune.

Wow, it's fascinating how a single HN comment can drive meaningful traffic to a project! I'm the author of Centrifugo, and I appreciate you mentioning it here.

Let me share a bit more about Centrifugo transport choices. It’s not just about supporting multiple transports — developers can also choose between bidirectional and unidirectional communication models, depending on their needs.

For scenarios where stable subscriptions are required without sending data from the client to the server, Centrifugo seamlessly supports unidirectional transports like SSE, HTTP-streaming, unidirectional gRPC streams, and even unidirectional WebSockets (this may sound kinda funny for many I guess). This means integration is possible without relying on client-side SDKs.

However, Centrifugo truly shines in its bidirectional communication capabilities. Its primary transport is WebSocket – with JSON or Protobuf protocols, with SSE/HTTP-streaming fallbacks that are also bidirectional — an approach reminiscent of SockJS, but with more efficient implementation and no mandatory sticky sessions. Sticky sessions is an optimization in Centrifugo, not a requirement. It's worth noting that SSE only supports JSON format, since binary is not possible with it. This is where HTTP-streaming in conjuction with ReadableStream browser API can make much more sense!

I believe Centrifugo gives developers the flexibility to choose the transport and communication style that best fits their application's needs. And it scales good out of the box to many nodes – with the help of Redis or Nats brokers. Of course this all comes with limitations every abstraction brings.

Hello, I am author of https://github.com/centrifugal/centrifugo. Our users can choose from WebSocket, EventSource, WebTransport (experimental for now, but will definitely stabilize in the future). WebRTC is out of scope as the main purpose is central server based real-time json/binary messaging, and WebRTC makes things much more complex since it shines for peer-to-peer and rich media communications.

What I'd like to add is that Centrifugo also supports HTTP-streaming – not mentioned by the OP – but this is a transport which has advantages over Eventsource - like possibility to send POST body on initial request from web browser (with SSE you can not), it supports binary, and with Readable Streams browser API it's widely supported by modern browsers.

Another thing I'd like to mention about Centrifugo - it supports bidirectional WebSocket fallbacks with EventSource and HTTP-streaming, and does this without sticky sessions requirement. I guess nobody else have this at this point. See https://centrifugal.dev/blog/2022/07/19/centrifugo-v4-releas.... Which solves one more practical concern. Sticky sessions is an optimization in Centrifugo case, not a requirement.

If you are interested in topic, we also have a post about WebSocket scalability - https://centrifugal.dev/blog/2020/11/12/scaling-websocket - it covers some design decisions made in Centrifugo.

Wondering whether coroutines may be a step towards async event-based style APIs without allocating read buffers for the entire connection. I.e. a solution to problems discussed in https://github.com/golang/go/issues/15735. Goroutines provide a great way to have non-blocking IO with synchronous code – but when it comes to effective memory management with many connections Go community tend to invent raw epoll implementations: https://www.freecodecamp.org/news/million-websockets-and-go-.... So my question here – can coroutines somehow bring new possibilities in terms of working with network connections?

Every time I read criticism of WebSockets it reminds me about WebSuckets (https://speakerdeck.com/3rdeden/websuckets) presentation :)

I am the author of Centrifugo server (https://github.com/centrifugal/centrifugo) - where the main protocol is WebSocket. Agree with many points in post – and if there is a chance to build sth without replacing stateless HTTP to persistent WebSocket (or EventSource, HTTP-streaming, raw TCP etc) – then definitely better to go without persistent connections.

But there are many tasks where WebSockets simply shine – by providing a better UX, providing a more interactive content, instant information/feedback. This is important to keep - even if underlying stack is complicated enough. Not every system need to scale to many machines (ex. multiplayer games with limited number of players), corporate apps not really struggle from massive reconnect scenarios (since number of concurrent users is pretty small), and so on. So WebSockets are definitely fine for certain scenarios IMO.

I described some problems with WebSockets Centrifugo solves in this blog post - https://centrifugal.dev/blog/2020/11/12/scaling-websocket. I don't want to say there are no problems, I want to say that WebSockets are fine in general and we can do some things to deal with things mentioned in the OP's post.

Hi! Could you please extend the meaning of word "global-scale" a bit? Does this only mean that users will connect to the nearest server or there are more tricks on backend to scale PUB/SUB? You are writing that "messages are optimized for speed across our global network" – could you also elaborate more what do you mean saying this?

Thanks for feedback, I thought the article left fully unnoticed here:) Go also has a small disadvantage for writing WebSocket apps – you don't have full control on read buffer reuse (since its got allocated for entire lifetime of connection) thus memory consumption per connection is more than it could be. Otherwise I am pretty happy with Go for this task.

I also don't really like new design of main repo page. Repo short description on the right – so inconvenient. BTW this is how it looks on mobile: https://i.imgur.com/SqV23WN.jpg - the most interesting things about repo usually located in first lines of README - now we can't see it.