News for hackers. Stuff that matters.
Re:Scaling Datastores at Slack with Vitess by yashap on Sunday December 06, 2020 @03:43PM (#25326446)
> Today, we serve 2.3 million QPS at peak. 2M of those queries are reads and 300K are writes.

Damn, even for a service as popular as Slack, that’s significantly more than I expected. Slack has ~12-13 mil DAUs, right? I assume at peak time of day, maybe 2-3 million actively using the product at the same time? If that’s a fair assumption, that’s roughly 1 MySQL query per second, per active customer - seems like a fair bit? I wonder if they do polling (instead of websockets)?

At my work we have roughly 1 order of magnitude fewer DAUs, but roughly 2 orders of magnitude fewer QPS. And we also have chat areas of the product.

Re:Scaling Datastores at Slack with Vitess by jhgg on Sunday December 06, 2020 @04:14PM (#25326708)
It is a pretty high QPS. I work on a chat service that is quite popular (even more popular than Slack) and looked at our QPS to our persistent data stores. It is less than an order of magnitude compared to slack's when you normalize for DAU. Really wonder what they're doing over there.
Re:Scaling Datastores at Slack with Vitess by derekperkins on Monday December 07, 2020 @01:18AM (#25329798)
Whatever the service, it isn't the platform that Slack is. I would guess that a huge percentage of their db traffic is from apps and custom integrations.
Re:Scaling Datastores at Slack with Vitess by karlding on Sunday December 06, 2020 @03:59PM (#25326561)
> Slack has ~12-13 mil DAUs, right?

As of September 2019, Slack reported 12 million DAUs [0]. Of course, that is pre-pandemic, and the only figures that Slack has provided regarding demand in 2020 are these tweets [1] from Slack CEO Stewart Butterfield back in March. In those tweets, it mentions that Slack was serving 12.5M simultaneously connected users.

[0] https://slack.com/blog/news/work-is-fueled-by-true-engagemen...

[1] https://twitter.com/stewart/status/1243000487365861376

Re:Scaling Datastores at Slack with Vitess by ahachete on Sunday December 06, 2020 @05:01PM (#25327036)
GitLab.com is serving around 300QPS at peak (see https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/).

While that's like 1/7th of Slack's traffic, it is served by a single PostgreSQL cluster.

That's the comment.

Re:Scaling Datastores at Slack with Vitess by yashap on Monday December 07, 2020 @01:28AM (#25329843)
May want to edit your post - I believe you meant 300K QPS, not 300 QPS :) But yeah, that’s very impressive to handle that much load without sharding! Though it is a cluster of 12 instances, each with 96 CPUs and 614 GB RAM - those are some seriously beefy machines.

Slack are getting 300K WRITES per second, though. Which does mean you have really no choice but to shard - read replicas are nice for scaling reads, but do nothing for writes, and that’s A LOT of writes.

Re:Scaling Datastores at Slack with Vitess by ahachete on Monday December 07, 2020 @12:33PM (#25334467)
Cannot edit any longer, but yes, that's a typo, it's 300K QPS.

Right now the cluster is only 8 instances, and the write-only traffic on the master spikes to 70K QPS.

So I agree with you --pretty beefy hardware, and it'd probably require sharding before reaching Slack's scale, but still quite impressive.

Re:Scaling Datastores at Slack with Vitess by conradfr on Sunday December 06, 2020 @05:54PM (#25327453)
Does that account for all those app/bots monitoring/CI/VCS/etc that "spam" channels?