Re:Lessons from 10 Years of Amazon Web Services
by jcrites on Saturday March 12, 2016 @12:43AM
(#11271566)
James has put out a lot of great material. He's given a talk series and wrote a conference paper called "On Designing and Deploying Internet-Scale Services" which is a trove of valuable design insights:
He writes some of the more practical and useful advice I've seen about how to run a successful business based on high-scale distributed systems.
One of his mnemonics that's stuck with me is the "Four Rs" of recovery-oriented computing: Restart, Reboot, Reimage, Replace (slide 6 in PDF). A human shouldn't be engaged to troubleshoot a problem until the platform has first tried in successing to restart the software, reboot the OS, re-image the machine, and finally replace the hardware entirely. After these auto-recovery steps have failed, only then is it time to engage a human. He describes the connection between these techniques and significantly lower operations costs.
Re:Lessons from 10 Years of Amazon Web Services
by jimbokun on Friday March 11, 2016 @04:42PM
(#11269751)
"Primitives not frameworks"
This is why Amazon beat Google in the "as a Service" space. Google provided a framework with AppEngine, but customers wanted the flexibility of the primitives offered by AWS.
Re:Lessons from 10 Years of Amazon Web Services
by nopzor on Friday March 11, 2016 @04:51PM
(#11269803)
I think that Amazon being first in 2005/2006 helped a lot, too. Google now provides "primitives" through GCE. I think Google realizes that the future is GCE not GAE.
Re:Lessons from 10 Years of Amazon Web Services
by vgt on Friday March 11, 2016 @05:19PM
(#11269966)
I think there's tremendous value still in GAE. To this day the fact that a company the size of Snapchat is able to operate on a PaaS, with virtually no ops people, is STILL a game-changer. The fact that it's still true 3+ years in is equally as impressive.
The narrative shifted away from PaaS for a number of reasons, but the value prop is still present.
(Disclaimer: work on BigQuery, a DBaaS, which is, in my opinion, is clearly the future)
Re:Lessons from 10 Years of Amazon Web Services
by nopzor on Saturday March 12, 2016 @07:56AM
(#11272666)
Sun Grid was more about batch processing across many machines, rather than individual VMs, wasn't it?
Plenty of people in the web hosting industry offered competitive x86 linux instances or dedicated servers, way prior to 2006.
I think AMZN deserves credit for delivering a price compelling option that delivered (1) elasticity (per hour billing) and (2) automation (API, available in minutes), among other things.
Re:Lessons from 10 Years of Amazon Web Services
by eranation on Friday March 11, 2016 @11:05PM
(#11271351)
Exactly
Offer IaaS first, the PaaS. Google had it backward.
While AWS does have a small vendor lock in too, in the end it's still recognizable , I mean it's doable for example to move an AWS architecture to open stack without a major code rewrite. If you built your web app on GAE and GWT however, your investment to move it out of Google will be much bigger.
Re:Lessons from 10 Years of Amazon Web Services
by rodionos on Friday March 11, 2016 @06:29PM
(#11270356)
You have to give full credit to AWS for inventing previously unknown pricing models. In some cases, it was actually the pricing model that created new use cases. Glacier and S3 comes to mind.
Re:Lessons from 10 Years of Amazon Web Services
by jeffbarr on Friday March 11, 2016 @11:27PM
(#11271400)
That's a very important observation. When I describe AWS to audiences I always make clear that it is a combination of a technology and a business model.
Re:Lessons from 10 Years of Amazon Web Services
by bluecmd on Friday March 11, 2016 @04:21PM
(#11269641)
So, with No SSH - how do you debug that one-off problem that is only on machine abc12? I'm not talking about mutating, I'm talking about attaching gdb to the process while it's handling requests. I'm talking about collecting CPU profiling information from production. Stuff like that.
Not terribly easy, but possible. I agree that "never have to SSH" is a bad target because there will always be edge cases, but for the majority of things it's possible to troubleshoot at the cluster level given immutable infrastructure; all the systems will be doing the exact same thing given an input. Reality shows that this is a bit idealistic, but if you start finding that e.g. A certain set of requests cause weird behavior, you can look at what makes those requests different, then set up some routing logic to send them to a debug farm (using something like https://github.com/Netflix/zuul ).
Re:Lessons from 10 Years of Amazon Web Services
by autotune on Friday March 11, 2016 @04:32PM
(#11269696)
You could probably have a central logging service like graylog or logstash that automatically collects that information and just sort through the logs that way.
Re:Lessons from 10 Years of Amazon Web Services
by api on Friday March 11, 2016 @05:41PM
(#11270070)
What if the problem is not diagnosable from logs?
In a perfect world production and development would be identical, all production issues would be easy to reproduce in development, and logging and error handling would be sufficient to diagnose any problem.
But what do you do when you have a problem that doesn't happen in a perfect world -- that you can't diagnose normally, doesn't occur in dev, shows nothing abnormal in logs, etc.? At some point you need the ability to inspect the actual thing. It's like diagnosing a problem with a bridge by doing experiments on a scale model replica. Doesn't always work.
Re:Lessons from 10 Years of Amazon Web Services
by autotune on Friday March 11, 2016 @07:38PM
(#11270656)
In that case I'd agree with you, and you should still allow for the possibly of logging in via ssh, while aiming to reduce that need and focus on automation as much as possible, or as much as it's appropriate and possible for your environment, rather.
Re:Lessons from 10 Years of Amazon Web Services
by autotune on Friday March 11, 2016 @05:18PM
(#11269951)
Launch a copy of that server with a new image (which I would hope is under a load balancer), update the sysctl.conf file template with a larger fs.file-max limit in whatever config management system you use for that instance, resume logging under freshly cloned instance.
Re:Lessons from 10 Years of Amazon Web Services
by dsp1234 on Friday March 11, 2016 @05:38PM
(#11270059)
How would you know to do that at all? The proposed issue is that the file handles are exhausted. Thus logging doesn't actually work, and thus you don't know anything about the problem except 'it's not working'. Your solution presupposed that you have figured out the issue, but the question is how do you find out about the problem at all. Without SSH/debugger/whatever, what is the process to inspect the system to find the problem?
Re:Lessons from 10 Years of Amazon Web Services
by devonkim on Friday March 11, 2016 @10:28PM
(#11271257)
After dealing with different enterprise systems in AWS for a few years now, I'd say that this is what AWS ELB feature to remove a system out of an ASG as part of its lifecycle exists for. If you aren't using similar horizontal scaling strategies, you have a disaster waiting to happen basically and basically deserve an outage you can't diagnose / RCA well. If you can't ssh in afterwards that means your provisioning for automation / instrumentation and you should stay be able to isolate attention to that part of an instance's lifecycle.
Re:Lessons from 10 Years of Amazon Web Services
by autotune on Friday March 11, 2016 @07:33PM
(#11270630)
Great question. I'd probably look into something like Sysdig Cloud that can collect that information through its own agent, or one of the tools mentioned by seanp2k2. I do also agree with his point on edge cases though, it should be the goal for the majority of your infrastructure while still allowing for edge cases.
https://www.usenix.org/legacy/event/lisa07/tech/full_papers/... (somewhat condensed)
Slides from a talk: http://mvdirona.com/jrh/talksAndPapers/JamesRH_AmazonDev.pdf
He writes some of the more practical and useful advice I've seen about how to run a successful business based on high-scale distributed systems.
One of his mnemonics that's stuck with me is the "Four Rs" of recovery-oriented computing: Restart, Reboot, Reimage, Replace (slide 6 in PDF). A human shouldn't be engaged to troubleshoot a problem until the platform has first tried in successing to restart the software, reboot the OS, re-image the machine, and finally replace the hardware entirely. After these auto-recovery steps have failed, only then is it time to engage a human. He describes the connection between these techniques and significantly lower operations costs.
This is why Amazon beat Google in the "as a Service" space. Google provided a framework with AppEngine, but customers wanted the flexibility of the primitives offered by AWS.
The narrative shifted away from PaaS for a number of reasons, but the value prop is still present.
(Disclaimer: work on BigQuery, a DBaaS, which is, in my opinion, is clearly the future)
Plenty of people in the web hosting industry offered competitive x86 linux instances or dedicated servers, way prior to 2006.
I think AMZN deserves credit for delivering a price compelling option that delivered (1) elasticity (per hour billing) and (2) automation (API, available in minutes), among other things.
Offer IaaS first, the PaaS. Google had it backward.
While AWS does have a small vendor lock in too, in the end it's still recognizable , I mean it's doable for example to move an AWS architecture to open stack without a major code rewrite. If you built your web app on GAE and GWT however, your investment to move it out of Google will be much bigger.
Amen. We couldn't agree more: https://boxfuse.com/blog/no-ssh
http://techblog.netflix.com/2012/06/scalable-logging-and-tra...
https://github.com/Netflix/vector
Not terribly easy, but possible. I agree that "never have to SSH" is a bad target because there will always be edge cases, but for the majority of things it's possible to troubleshoot at the cluster level given immutable infrastructure; all the systems will be doing the exact same thing given an input. Reality shows that this is a bit idealistic, but if you start finding that e.g. A certain set of requests cause weird behavior, you can look at what makes those requests different, then set up some routing logic to send them to a debug farm (using something like https://github.com/Netflix/zuul ).
In a perfect world production and development would be identical, all production issues would be easy to reproduce in development, and logging and error handling would be sufficient to diagnose any problem.
But what do you do when you have a problem that doesn't happen in a perfect world -- that you can't diagnose normally, doesn't occur in dev, shows nothing abnormal in logs, etc.? At some point you need the ability to inspect the actual thing. It's like diagnosing a problem with a bridge by doing experiments on a scale model replica. Doesn't always work.
Instead of ssh vs. no-ssh debate, how about:
* Use SSH AllowGroups to limit logins to the development manager/lead or on-call engineer
* with a policy that for each time you have to call them beyond N times per week, you have to buy their team a round of drinks of their choosing?
Deal?