Re:Former Tesla Programmer's anecdotes about problems
by ckdarby on Friday November 06, 2020 @10:04AM
(#25008064)
Sounds like every company for those who have spent time in the software world at a large enough company.
Example of some of the stuff I've seen at various places:
- Team A self ddos Team B taking down a critical platform that stopped the entire company from processing orders.
- Bug lead to everyone's password being reset to QA value and restoring backups would have taken too long. Any customer who complained about access was told support had reset their password just now to test123 but in reality every customer's password had been reset to that.
- Another scenario where some QA values got ran on a production environment and some settings and configs had to be manually restored as it was a third party service it got applied against. Halted nearly whole company.
- Bug miscalculates how to charge users and basically drains most people's bank accounts.
- Accidental DNS rebinding exposing CDN traffic to another company and took them down.
- Dev mistake bypassing a endpoints on the CDN for caching and throttling rules. Nearly $1M mistake before anyone even noticed.
- Accidentally using another S3 bucket template for another S3 bucket and not noticing it had versioning on. Program that dumped it's state for restoring to this versioning bucket had +450TB of state versioned. It didn't care about the history of state just the present state for restoring. Noticed only by mistake when cleaning up somethings.
And many other stories, things happen, you learn, you improve and it is really a risk vs reward calculation on the business side.
Re:Former Tesla Programmer's anecdotes about problems
by raiyu on Friday November 06, 2020 @12:48PM
(#25009537)
That's doubtful, companies get harder to manage at scale not easier.
And real long term issues like this are usually fixed through massive culture changes, leadership changes, followed by extremely heavy lifting by ICs.
Just look at Microsoft evolving from Windows to the Cloud, to Azure.
Two years, when you are dealing with a ton of software and hardware and a CEO who has somewhat of a "god" complex - it's unlikely that things have changed all too much.
More likely, if they decided to retool for the Model 3 and build from the ground up, yes, but that would add time delay, and Elon always feels like he's racing against an internal clock in his head so it's doubtful.
Example of some of the stuff I've seen at various places:
- Team A self ddos Team B taking down a critical platform that stopped the entire company from processing orders.
- Bug lead to everyone's password being reset to QA value and restoring backups would have taken too long. Any customer who complained about access was told support had reset their password just now to test123 but in reality every customer's password had been reset to that.
- Another scenario where some QA values got ran on a production environment and some settings and configs had to be manually restored as it was a third party service it got applied against. Halted nearly whole company.
- Bug miscalculates how to charge users and basically drains most people's bank accounts.
- Accidental DNS rebinding exposing CDN traffic to another company and took them down.
- Dev mistake bypassing a endpoints on the CDN for caching and throttling rules. Nearly $1M mistake before anyone even noticed.
- Accidentally using another S3 bucket template for another S3 bucket and not noticing it had versioning on. Program that dumped it's state for restoring to this versioning bucket had +450TB of state versioned. It didn't care about the history of state just the present state for restoring. Noticed only by mistake when cleaning up somethings.
And many other stories, things happen, you learn, you improve and it is really a risk vs reward calculation on the business side.
And real long term issues like this are usually fixed through massive culture changes, leadership changes, followed by extremely heavy lifting by ICs.
Just look at Microsoft evolving from Windows to the Cloud, to Azure.
Two years, when you are dealing with a ton of software and hardware and a CEO who has somewhat of a "god" complex - it's unlikely that things have changed all too much.
More likely, if they decided to retool for the Model 3 and build from the ground up, yes, but that would add time delay, and Elon always feels like he's racing against an internal clock in his head so it's doubtful.
Kinda like Wikileaks for horror tech stories. Anyone remember it?