How Discord stores trillions of messages

Re:How Discord stores trillions of messages by ryanjshaw on Sunday September 29, 2024 @02:22AM (#41685311)

Okay but this is where I get confused. Why does Discord need a single database system when discord servers are independent, right?

And the volume of traffic per Discord server must be human-processable or what would the point be? A Discord server doing 800k writes per second makes no sense.

So why not a RDBMS per Discord server, and if you want to ship all that out to a warehouse for analytics you do that as a separate problem?

Or is it that spinning up a Postgres instance per Discord server ends up being significantly more expensive than these mega distributed database systems?

[ Parent | on HN ]

Re:How Discord stores trillions of messages by jhgg on Sunday September 29, 2024 @02:45AM (#41685376)

There are ballpark of a few hundred million discord servers... do you really want to run that many Postgres instances? And even so what do you do about DM/GDMs? Easier to just run one big mega cluster for messages.

[ Parent | on HN ]

Re:How Discord stores trillions of messages by ryanjshaw on Sunday September 29, 2024 @02:59AM (#41685426)

Okay so the latter then - economies of scale. Surprised to hear that few hundred million figure - I thought it'd be 1/10th of that at most! Wow.

Although I did expect there'd be a very long tail, and you might choose to host a bunch of servers on a single RDBMS, at that scale yeah it wouldn't solve much.

Thanks for coming back to me, appreciate it.

[ Parent | on HN ]

Re:How Discord stores trillions of messages by Drew_ on Sunday September 29, 2024 @12:21PM (#41688412)

Apple kind of does something like this with iCloud however their per user "databases" are only virtual:

https://news.ycombinator.com/item?id=39028672

[ Parent | on HN ]