News for hackers. Stuff that matters.
Re:What I Wish I Had Known Before Scaling Uber [video] by adekok on Wednesday September 28, 2016 @10:42AM (#12598006)
I suspect that the issue isn't the engineers. It's the changes.

A system running just fine will usually continue to run just fine. Until it's changed.

Re:What I Wish I Had Known Before Scaling Uber [video] by Tenhundfeld on Wednesday September 28, 2016 @02:33PM (#12600090)
I hear what you're saying, in theory. In practice, a system is always changing – regardless of whether developers are deploying code changes.

Some events (or "changes") I've seen in last year that caused a system to stop running fine:

- analyst ran query that locked reporting DB, which had ripple effects

- integration partner "tweaked" schema of data uploaded to us

- NIC started dropping packets

- growing data made backup software exceed expected window, locking files

- critical security patch released (didn't cause a problem exactly, but did change the system from having no known security holes to having a big one)

On and on. So, yeah, I'm not disputing the idea that changes are often the source of problems. I'm just saying that any moderately complex system is constantly encountering new events whether or not the developers are making changes.

This isn't a rebuttal of your statement so much, as it is rebuttal of a common attitude I see in business folks. A lot of non-technical executives seem to have the mentality that software is "done" after it's built, which is naive IMO.

Active systems require active maintenance. You can avoid a lot of problems with intelligent architecture and robust instrumentation/monitoring, but at the end of the day systems rot and will eventually stop running fine.

And even if your perfectly planned and architected system runs in total isolation on a private server where security isn't a major concern, you'll still build up technical debt if you aren't routinely upgrading major libraries, etc. You'll get to a point where you want to implement some feature, and while there are 3 excellent OSS libraries for your platform to accomplish it, none of them is compatible with your 5 year old version.

You probably know all of this and might even agree with most of it. I just had a visceral reaction to the idea that "a system running just fine will usually continue to run just fine," and felt compelled to respond. :)

Note: most of my experience is with public or enterprise web applications. I imagine other types of systems have other problems.

Re:What I Wish I Had Known Before Scaling Uber [video] by fail2fail2ban on Wednesday September 28, 2016 @01:10PM (#12599393)
Or until the power is cycled. That's why it's best to do the crazy monkey or whatever it's called, walk into the production offline datacenter, online for testing (you do have geographically redundant datacenters right?) and pull out a half dozen random NIC cables. You can simulate with software tools but nothing beats the real test.
Re:What I Wish I Had Known Before Scaling Uber [video] by softawre on Wednesday September 28, 2016 @01:17PM (#12599458)
> crazy monkey

Literally lol'd. It's "chaos monkey".

Re:What I Wish I Had Known Before Scaling Uber [video] by jsprogrammer on Wednesday September 28, 2016 @01:27PM (#12599531)
Chaos monkey is a simulation, I believe.
Re:What I Wish I Had Known Before Scaling Uber [video] by bch on Wednesday September 28, 2016 @03:40PM (#12600638)
It's not a simulation, it's real-world production technology.

https://en.wikipedia.org/wiki/Technical_details_of_Netflix

Re:What I Wish I Had Known Before Scaling Uber [video] by s_kilk on Thursday September 29, 2016 @03:44AM (#12603833)
I think they meant that Netflix Chaos Monkey is a "simulation" of the real-deal, which is sending a shaved-ape into a physical server room and pulling out physical network cables. I didn't read it as derogatory.
Re:What I Wish I Had Known Before Scaling Uber [video] by bch on Thursday September 29, 2016 @01:32PM (#12607140)
Oh perhaps they did mean shaved-ape simulator. I read none of it as derogatory, and hopefully my response didn't suggest otherwise.
Re:What I Wish I Had Known Before Scaling Uber [video] by rrdharan on Wednesday September 28, 2016 @11:28PM (#12603111)