Re:Heartbeats in Distributed Systems
by zbentley on Thursday November 13, 2025 @06:05PM
(#45921903)
> The only way I can see it being a lot of work is if your nodes are very small
Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often.
> you're going to hold onto the partial heartbeat for like 2 seconds
And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats.
Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.
Which is a totally valid use case for a huge-nodecount cluster! Geo distribution or similar use cases employ this pattern often.
> you're going to hold onto the partial heartbeat for like 2 seconds
And if a region goes down such that a large number of nodes are tarpit-heartbeating at the same time? Your cluster can fail to accept real traffic because it’s clogged with heartbeats.
Like sure, generally 2krps should be fine. But in clusters this large it is totally reasonable to tune heartbeat behavior. I’ve seen plenty of real distributed system failures along these same exact lines.