How Discord Stores Billions of Messages Using Cassandra

Re:How Discord Stores Billions of Messages Using Cassandra by yawn on Thursday January 19, 2017 @08:38PM (#13441108)

> These kinds of write-ups offer valuable insight

I agree completely. It is frustrating that no decent books have been written regarding scaling architectures/strategies with current tooling. One has to scavenge various blog posts to try and discover ideas that might help solve their growth issues. I would love to see a book that covers scaling for app servers, RDBMSes, NoSql dbs, using queues/messaging effectively, etc. Failing that, I'd like to see something like Scalers at Work (a la Coders at Work) which would interview different devs who had to solve scaling issues.

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by jjirsa on Thursday January 19, 2017 @11:04PM (#13441667)

Look for local meetups - in Seattle there's "Seattle Scalability" which is great for this sort of thing (and highscalability used to be great for this, too).

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by roastbeef on Friday January 20, 2017 @06:06AM (#13442923)

is discord in seattle?

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by jjirsa on Friday January 20, 2017 @11:55AM (#13445080)

No, but it's where I live, and I was using it as an example of a non-silicon-valley city where you can find scalability related meetups.

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by wsh91 on Thursday January 19, 2017 @10:47PM (#13441609)

This isn't exactly what you want, but you might find both of the following books helpful.

With respect to using queues/messaging: http://www.enterpriseintegrationpatterns.com/

And with respect to understanding this stuff in general: http://dataintensive.net/

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by niftich on Thursday January 19, 2017 @11:25PM (#13441720)

Not the parent, but the linked material is more foundational than the subject matter raised in the post. There is in fact an appreciable lack of good, battle-tested, non-secret, sometimes-but-not-necessarily anecdotal public info about the part of the design process where you have a working system doing fairly okay, but you know you're inches away from a very unpleasant wall. On fire [1].

It doesn't help that distributed systems are a dark art, that many open source and free-to-use tools that developers have access to gate the HA/clustering features behind steep pricing (though I sympathize it's one of the few effective ways to make money in open source), and that expertise with scaling is very often a competitive advantage.

[1] http://www.slideshare.net/iammutex/scaling-instagram/25-404i...

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by wsh91 on Thursday January 19, 2017 @11:30PM (#13441739)

Totally fair. :) Parent just mentioned using queues/messaging effectively, and EIP is arguably the gold standard for that.

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by di4na on Friday January 20, 2017 @02:20AM (#13442206)

http://shop.oreilly.com/product/mobile/0636920024149.do

The first part is mainly about erlang and the choices they made. But the last part is not at all specific to erlang and walk you all the way through all decisions to take to build that type of architecture.

[ Parent | on HN ]

Re:How Discord Stores Billions of Messages Using Cassandra by threeseed on Friday January 20, 2017 @05:10AM (#13442715)

I've always subscribed to the HighScalability RSS feed: http://highscalability.com

[ Parent | on HN ]