Most applications are simply cloned from existing On-Premises Infrastructure because every project have a 2 weeks deadline and every single stakeholder just wants to stick it in. Patterns like Auto-Scaling etc is just a far fetched dream.
Supporting such Infrastructure where every server is a pet trying to survive in a Slaughter House is a deeply painful.
People of this kind are extremely conservative, because they are not in a position to fix major issues if something goes wrong--esp new architecture. So, they want to lift as is to aws.
A decade ago, I was involved in physically moving three solaris servers that were running un-supported BEA weblogic instances. When I powered on these three servers on a different rack, weblogic cluster failed. No IP change, just moved machines to free up racks in one row. And the people who developed the app, and the dev architect behind it were so mad at us. In the end, they gave up on that app.
That's how it goes.
For example stateless web component is a primary requirement to allow servers to fail as well as auto scale. Most enterprise applications I have seen falls flat on face.
I have seen enough cases in which customer endlessly debates why his snowflake of a web server went down and expects a RCA with god knows what details.
It's a terrible idea, but cloud providers love it because they end up getting paid a lot of money for idle capacity.
I said in another post that I was a dev lead for a company that decided to move to AWS. After dealing with “consultants”, realizing how shallow their knowledge was after I started learning about AWS and seeing how much we were paying them, I changed my whole career focus.
I got another job at a small company that was running completely on AWS who needed some “adult supervision”, self demoted to an individual contributor and I’m getting real world AWS experience so I can become an overpaid consultant who actually can consult with developers, devops, and infrastructure departments.
I am "lucky", I think, as my employer has already been through some very painful experiences (and learned nothing, apparently) - and they seem content to let me do whatever the fuck I want, as long as I sufficiently song-and-dance them. And yes, I've made some mistakes along the way. But I've also delivered results I didn't even think were going to be possible when I started. Advice I'd give to people walking this path:
1) You begin in an emergency. Everything is always an emergency. Because production is down. And you will always be wondering why your customers just don't drop you. But do take a moment to establish baseline metrics. Some way to measure how much you're getting, performance-wise, in terms of uptime, records processed, users, customer calls per day, cloud costs. Whatever you can pull out of your ass for minimal effort. Get SOMETHING as a baseline. As you go along, if you miss a deadline, nobody will question you if you can show progress in your metrics against that baseline.
2) The theoretical ideal model web apps may perform radically differently when you start monkeying with the architecture. If the original developers are gone, if the code is more than 5 years old, it may not be feasible to debug the squirrelly behavior you see when you change from a traditional proxy to say, an AWS load balancer. You can burn a shit ton of time trying to troubleshoot things like this. Slapping a band aid on top of 10 years old band aids is not always the right thing to do. But sometimes it's the best approach to delivering results that keep people convinced to fund your effort.
3) even if nobody else in your company has a fucking clue what you're doing, or understands it; document everything.
Get them on the cloud, then they can have the 'option' to start utilizing pure cloud type services, and migrate over.
Just reproducing their current setup on the cloud in many ways makes perfect sense.
Remember that for most companies, IT is just IT - it's an enabler, but not a huge differentiator. What 'works works', and tech change is expensive and risky.
Going to the cloud and shifting paradigms is two big steps, they can be taken one at a time.
But yes - any HN reader would be kind of 'shocked' to walk into almost any company in the world and see how 'high tech' really works. It's a valuable lesson in perspective, though.
Legacy apps asummes a certain degree of reliability from the underlaying infrastructure. This assumption falls flat on its face in the cloud. Netflix wouldn't have needed to develop Chaos Monkey if this wasn't the case.
Secondly leap to cloud native never happens because no one wants to take the pain once it works even barely. Any issues can be pushed onto Infrastructure team. It's pure and simple risk avoidance.
So you're tested on your understanding of the platform but not on your actual competences on it.
I've seen this in other areas however: people getting LPIC certifications and failing to run a proper grep.
In my experience, Red Hat is one of the few tech companies that get the certification program right by using an hands-on approach: either you actually are capable of performing tasks at a certain level, or you get not certification at all.
There are many professionals certified on AWS that aren't really able to perform properly on that platform, for example. Add the fact that plain old system administration I'd dying (less and less needed with the cloud and things like ansible)... And here we are.
The AWS Associate certs are similar to what you describe, particularly the "Solutions Architect" which is IMO common sense for a systems admin. The "Developer" one, though, requires memorizing some trivia around APIs and limits (eg., SQS queue sizes and stuff like that).
However, the AWS Professional certs are said to be MUCH harder, and while multiple choice, require reading a lot of scenario information quickly, and then selecting more than one answer (sometimes not being told how many answers). I haven't taken it but I expect to take the DevOps Professional exam at some point. I could be wrong, but everything I've seen so far, including sample questions, suggests it requires a fair bit of hands-on experience.
However, where everything I said becomes false is around the culture of answer dumps. I'm pretty sure there are dumps sold out there for cheaters to purchase. It's a pretty silly thing to do, especially because you'll be fired if you demonstrate incompetence in the field, but it does undermine the multiple choice test mechanism for certifications.
I would never try to get a pure AWS job with just theoretical knowledge.
From what I can tell, AWS recently offloaded their certification program onto a third party, so I doubt it will improve anytime soon.
https://www.cncf.io/certification/training/
(Disclosure: I helped develop the exam.)
(Also heard your interview on Software Engineering Daily the other day, very informative)
And, thank you. I've been a listener of Jeff's for a while and enjoyed our chance to chat. https://softwareengineeringdaily.com/2019/01/14/kubernetes-i...
* what do you mean who installs patches? I think our cloud provider does.
* (same, but for "firewall" or "policy")
* "I think they turn them off automatically if they're idle"
and so on. The "devops" hype has put people who think nothing of operations at all in charge of something they don't understand.
It tells me whether they have bothered to learn how to do stuff in the cloud or are just applying their old patterns to the cloud (at that point, they should just have colo/managed servers - they would save money, but usually the driver for the cloud came from the C-Suite in those situations - either way some training and a cost cutting project usually pays for itself in those situations).
This always kills me; the worst part is that keys are easier to use after you take 5 minutes to set them up once!
I’ve been using C# and MySQL and SQL Server for years. I’ve only done APIs in Node and Python with lambda. The on,y web server I know well is IIS.
I'm normally pretty certification averse (I don't even really value my undergraduate degree much), but I'm thinking of getting the AWS Solution Architect cert, because it seems like the material out there for learning that stuff is genuinely useful (and once you've learnt the material, you may as well get the certificate).
Associate's cert is mostly memorization and high level, but doesn't qualify you for being a trusted individual to do the job. It's good to make you aware of the right way to build things in AWS.
For $2/month you can make a static website, posted to S3, with a CloudFront front end and web hook to lambda to trigger content refreshes on specific Git actions.
I'd hire someone who did this with no professional experience with or without certification for an entry level position. That personal interest and drive is worth 1-2 years experience in my book... And I don't look for more than 6 months experience in entry level (I offer internships/ hourly for that group)
The tricky part seems like it would be dealing with tens-to-hundreds-to-thousands of machines which should automatically deploy and scale. And things like databases, which might also need to scale up/out at some point, and ideally should do without downtime.
That's much harder to try out on a personal project (due to the cost, but also due to the lack of genuine need shaping the implementation)...
I have a client that does $50M a year on a SAAS product using only two EC2 isntances, but 1 of those instances is a developer/beta/test server. Only 1 is used to manage sales across the entire world. They rely heavily on Lambda and SNS & SQS as well.
Another client operates eCommerce at around $80M a year revenue using 3 EC2 instances. One of those is a test/dev server and the other 2 are used to distribute load and for redundancy with a load balancer sitting on top.
I have 2 other companies that are smaller that generates about $2M and $5M a year respectively that both run a static site hosted on S3, using Lambda for deploys and cloudfront for distribution like mentioned above.
Point being, you would be surprised how low key many company's AWS deployments are.
---
If I could pin one skill that will stand you head and shoulders above the rest among cloud architects, it is to learn Serverless functions. In this AWS world this is "Lambda". Microsoft calls it "Azure Functions" and Google Cloud Platform calls it "Cloud Functions". If you can write serverless functions AND are AWS certified, that seems to be the hottest thing that I have seen.
It especially confuses me that people want people to "learn" serverless functions. Serverless functions are just like normal functions, except with a library to use for communicating with the outside world!
If I was hiring, then I'd be asking people about VPCs, IAM roles and Security Groups. Those seem to be the complicated bits that you need to understand to do pretty much anything. The service-specific APIs seem relatively simple...
There is a huge amount of IT work that never gets to that scale, and of the IT projects that get there, lots don’t actually need to be.
You must be dealing with exceptional companies who have application which need auto scaling. And need alone is not a sufficient condition, application should support it too.
Most application are simply lifted from legacy with no code changes and serves at-most a few hundred users at max and will crash and burn if autoscaled.
I have two or three ASGs up with a minimum and maximum of two and an HTTP health check that will kill an instance if the health check doesn’t pass and bring up another one.
I also have applications that autoscale based on queue sizes.
Scenario I am referring to involves application team closely guarding their servers like babies which means they are married to server names and IP addresses. These are cloned from on-premises and load balancing is just sticky sessions going to one node with other one sitting idle until first one fails.
First group is rare.
I wouldn’t go that far though. I would probably use something like Consul to do a local health check first and restart the app if it dies and then do the target group/health check as a second level check.
Still, it's interesting because some of it requires that you actively work against AWS to achieve (e.g. publicly accessible RDS). Could be that they gave up on, say, security groups and just found that to be the easiest way to "get it to work" (a frequent overriding directive).
In any case, I'd put some of it on Amazon too. Their documentation always struck me as piecemeal and task-oriented; lacking a sense of overall architecture or design.
But, I suppose that's what certifications are for.
Walk in and show a list of externally visible problems (in a way that clearly does not suggest "hacking"), and propose a project to get them correct.
Amazon, Google, Microsoft and others make it simple to get going but the devil is in the details when it comes to building it right, safe and ready for production. Just a few examples:
• Putting RDS SQL Servers on a public IP with no protection
• No templating of servers so if it disappears you have no record of how to get a new one running again
• Servers with SSH password authentication turned on with passwords of “Password”, because SSH key auth was “too hard for our devs”
• No backups because “it’s in the cloud, isn’t the cloud backing it up for me, like iCloud?”
I have an AWS Solution Architect Professional certification and agree you could get a cert and still not be qualified to run or design much on AWS but it does put you ahead of much of what I’ve seen out there.