Slack’s Incident on 2-22-22

Re:Slack’s Incident on 2-22-22 by Robin_Message on Wednesday April 27, 2022 @02:40AM (#31177020)

I am confused by the query that had the problem. Specifically, I am confused by why the sharding is done by user id.

Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second. That feels to me like it could be served by a single master DB. It feels like that would be a better way to shard.

Major downside is the major difference in shard sizes so some management/migration might be needed but it seems doable to me.

Certainly it feels naively that scaling should be easy due to the way slack instances are independent (unlike say Twitter).

[ Parent | on HN ]

Re:Slack’s Incident on 2-22-22 by sophiebits on Wednesday April 27, 2022 @02:53AM (#31177066)

Workspaces are not completely independent, such as for the Slack Connect feature. Some more details here: https://slack.engineering/scaling-datastores-at-slack-with-v...

[ Parent | on HN ]

Re:Slack’s Incident on 2-22-22 by Robin_Message on Wednesday April 27, 2022 @05:28AM (#31177981)

Thanks! Indeed, the variability of size and usage of each instance was the big issue, and then doing features that crossed instances meant they'd be crossing shards whatever they did, so it made sense to fix the variability issue.

(I'm also surprised/reminded how fast Slack grew and how quickly it became effectively ubiquitous — I think every company I've contracted for in the last five years has used Slack).

[ Parent | on HN ]

Re:Slack’s Incident on 2-22-22 by iamcal on Wednesday April 27, 2022 @12:02PM (#31181748)

> Even the largest Slack instance probably has under 100,000 users and less than 1000 peak messages per second.

This is not true, by an order of magnitude.

[ Parent | on HN ]