Re:Uber Freezes Hiring of U.S. Tech Staff as It Seeks to Cut...
by geofft on Saturday August 10, 2019 @11:33AM
(#20662609)
Slack does (or did?) exactly this, I think. Each Slack workspace is its own deployment: its own web server, its own database, etc. This is loosely why accounts aren't shared across workspaces, you need to open a new tab per workspace, and so on.
What this forces you to do is to get really good at configuration management and orchestrating deploys, so that when something goes wrong, you can fix it for all your customers at once. You can't treat any single instance as a thing that it's meaningful to log into (you might still want to log into things to debug problems after a customer report, but making changes is infeasible, and ideally you notice problems before the customer report so you have to be aggregating logs/events/metrics). So many of the same maturity lessons as required for building a single scalable instance continue to apply to you. The distinction is that you don't have a giant distributed system, and you don't have the corresponding operational burden of a giant distributed system, so you avoid outages like https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... . (To be clear - this isn't meant to shame Monzo, that was a genuinely difficult problem and a genuinely good incident response. But there are advantages in not having a single company-wide database that runs everything from your core business function to your customer support chat.)
One of the engineering lessons I've been learning of late is you really want to avoid distributed systems wherever possible. They're cool, but the amount of engineering / operational effort required to get to a certain level of consistency or availability is way more than the effort required to get to that level of availability of a non-distributed system. It's fun as an engineer to solve hard problems, but it's better for the business to not have hard problems in the first place.
What this forces you to do is to get really good at configuration management and orchestrating deploys, so that when something goes wrong, you can fix it for all your customers at once. You can't treat any single instance as a thing that it's meaningful to log into (you might still want to log into things to debug problems after a customer report, but making changes is infeasible, and ideally you notice problems before the customer report so you have to be aggregating logs/events/metrics). So many of the same maturity lessons as required for building a single scalable instance continue to apply to you. The distinction is that you don't have a giant distributed system, and you don't have the corresponding operational burden of a giant distributed system, so you avoid outages like https://monzo.com/blog/2019/09/08/why-monzo-wasnt-working-on... . (To be clear - this isn't meant to shame Monzo, that was a genuinely difficult problem and a genuinely good incident response. But there are advantages in not having a single company-wide database that runs everything from your core business function to your customer support chat.)
One of the engineering lessons I've been learning of late is you really want to avoid distributed systems wherever possible. They're cool, but the amount of engineering / operational effort required to get to a certain level of consistency or availability is way more than the effort required to get to that level of availability of a non-distributed system. It's fun as an engineer to solve hard problems, but it's better for the business to not have hard problems in the first place.