AWS mistakes to avoid

Re:AWS mistakes to avoid by devonkim on Saturday December 26, 2015 @10:17PM (#10796262)

Most of the reasons to go with AWS as a major strategic decision are to avoid paying for 3-4 (or perhaps... 40+) ops engineers to run and manage so many things that really don't add much business value in themselves and that you could hold to some form of an SLA (especially with enterprise support accounts) and recoup some costs if things came down to it. Furthermore, engineering a lot of AWS services would cost you developer time as well.

I consult / contract for a Fortune 50 customer that pays an insane amount of money to AWS per month and it seems ridiculous what they're paying for and what they get. But do you know what's even more ridiculous? Paying three times as much but getting even worse reliability and a massive amount of time wasted while waiting weeks to provision servers. One group internally used to have massive outages daily resulting in needing to hire a dedicated team of about 6 just to handle that load of production incidents (terrible reliability also resulting in loss of revenue too) and since migrating their software with even the most bone-headed SPF-everywhere AWS based software architecture haven't had more than one or two per week for outages now. The cost benefits get better the more incompetent / incapable your internal IT organization is. The amount of wasted resources due to internal bureaucracy, legacy, and working with companies that don't really practice any form of technology at scale has been an incredible amount of savings for my customer.

And knowing how much my customer relies upon AWS support for the most menial of tasks (primarily a cultural thing with how they treat their vendors as well as internal resources) I am sure that nobody but massive companies that deal with external bureaucracies effectively could deliver the size and kind of support team to handle the volume of support requests generated by my customer for the most ridiculous of tasks (I've seen executives file AWS support requests to reboot VMs at 3 am because the bureaucracy is extremely brittle here and reflects into the software systems reliability). We may throw millions / month at AWS, but Amazon has to hire at least a few dozen engineers and several account managers just to handle support alone which I might argue could be a net loss to handle the customer.

Most vendors that work with my customer have a very hard time working with them because they approach sourcing quite similar to how Walmart does and abuses them to the point where the vendors abuse their employees.

[ Parent | on HN ]

Re:AWS mistakes to avoid by drieddust on Monday December 28, 2015 @04:41AM (#10800513)

"I've seen executives file AWS support requests to reboot VMs at 3 am"

Isn't this falls outside amazon's responsibly area?

Atleast the documentation I read and a few aws promotional talks I attended seems to suggest the contrary.

Or I have understood completely wrong and amazon is now also providing managed IT services?

[ Parent | on HN ]

Re:AWS mistakes to avoid by devonkim on Monday December 28, 2015 @01:31PM (#10802539)

There are some legitimate cases where AWS support is the right place to issue a request. Just the other day I had a VM go down and I issued a request to stop the instance and it just hung there with no valid way to keep shutting it down. Turns out AWS support is the only way for most users to get an instance on a host that suddenly goes down while stopping your VM to be correctly stopped because the underlying API call does not go through and it seems like a small critical section of sorts.

But when you pay AWS so much they are pretty beholden to your requests and can wind up becoming a lot like an MSP. I've spent countless hours with those poor guys waiting on a traceroute failure or for mtr or sat to give some anomalous event that everyone suspects is something wrong in AWS when I've insisted AWS is almost never at fault because our incompetence is almost certainly the problem.

The AWS support team's worst engineer is probably at least in the top 25% of our support engineers I'd argue so I'm sure if my customer had the option they'd want to pay to have them troubleshoot our own internal networks but that's what my team is for I guess.

[ Parent | on HN ]

Re:AWS mistakes to avoid by late2part on Saturday December 26, 2015 @10:29PM (#10796292)

"The cost benefits get better the more incompetent / incapable your internal IT organization is. "

Yes. There will always be a less stupid solution out there that is not as bad as the bad solution.

Or, you could fix IT. But that takes thought.

Easier to give money to a vendor you can blame...?

[ Parent | on HN ]

Re:AWS mistakes to avoid by devonkim on Saturday December 26, 2015 @11:36PM (#10796421)

Fixing completely dysfunctional F500 IT organizations is something that many, many people have tried across completely different company cultures with maybe a handful of successes I've heard of. Leading huge organizations (especially ones damaged by previously failed leadership efforts - think how attempts to refactor a legacy codebase go and multiply by an order magnitude in number of variables) is very hard and because large organizations tend to have so much capital working for them, IT management failures create a large opportunity (in theory) for optimization by others. In practice, consulting for organizations that don't want to fundamentally change how they do X (or think $ must solve everything and leadership becomes so fragmented into non-communicating middle managers) while wanting large changes to X are the equivalent of an obese person wanting to lose weight without changing their diet or exercising. Yet we know there's a huge "fitness" industry with large profits being made off of those that do not use their products and subscriptions effectively.

[ Parent | on HN ]

Re:AWS mistakes to avoid by dev360 on Tuesday December 29, 2015 @12:07PM (#10807612)

Change is tricky. What you said reminded me of an interesting tweet I came across: https://twitter.com/JICare/status/662339652825780224

[ Parent | on HN ]

Re:AWS mistakes to avoid by JabavuAdams on Saturday December 26, 2015 @11:05PM (#10796360)

This is a fairly naive way of looking at it. Often it's easier to deal with an external organization than to fix your own organization's behaviour.

In a lot of pathologically-managed companies or divisions, there is effectively zero chance of an employee being able changing the pathology.

[ Parent | on HN ]

Re:AWS mistakes to avoid by devonkim on Sunday December 27, 2015 @08:03AM (#10797322)

I have heard of a few poorly managed companies pulling out of ruin by using consultants and third party labor to supplement their collapsing workforce. But I've seen poorly managed companies on occasion fire their executives and stabilize as well. People love to spend 7 figures on consultants and refuse to take their advice in favor of suck-ups that recommend that they just slightly change what they're doing or just sit in silence and continue to bill out. This is why I stopped consulting / contracting in the federal government space - nothing I do even if I was a company founder can fundamentally fix even a tiny fraction of the deep flaws. People got rich off of manipulating contracts to do as little as legally possible while the best at delivering were cut because they actually finished the job right. Reward for failure is a system that exists plenty in the private world, but only in public sector is it institutionalized.

[ Parent | on HN ]

Re:AWS mistakes to avoid by JabavuAdams on Sunday December 27, 2015 @02:07PM (#10798367)

This sounds about right. This is why I don't believe in the "industry is more efficient than government" dogma. Large organizations of people are generally inefficient unless extraordinary care has gone into the engineering and maintenance of the organization.

[ Parent | on HN ]

Re:AWS mistakes to avoid by mailshanx on Sunday December 27, 2015 @11:59AM (#10797967)

Are there any particularly memorable situations in your consulting career that illustrate the political realities of such organizations? Would be really awesome if you can share any :)

[ Parent | on HN ]