News for hackers. Stuff that matters.
Re:AWS down again? by koprulusector on Friday December 10, 2021 @09:31AM (#29510246)
I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work.

Seriously, those commenting “oh boy! Time to rethink this whole cloud thing!” You’re either so new to this stuff to have no experience to remember the days before cloud, you’re trolling because you’re high on nostalgia remembering the good ol’ days (perhaps with some “member berries” https://youtu.be/mPs8-ZBSjok), or you’ve just straight up forgotten what it takes to build and run your own infra from the ground up.

And to those who happen across this comment before posting your own: please contribute something to the discussion beyond the above. Thanks.

Re:AWS down again? by gerbilly on Friday December 10, 2021 @09:45AM (#29510423)
Well at least you said please, however:

I have built and run my own infrastructure from the ground up, and I've been made to transition to the cloud.

The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead.

Some of us old timers argued against this only to be shouted down by people who don't even understand TCP/IP.

The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design.

Today we'd just cruft together an MVP, solve only the interesting parts (or more likely the easy parts) and then move on, letting dashboards which lie to cover it up.

Many of us have been taking shots at those 'big boys' since the start of this trend.

Now that because of recent events and we have a chance to be heard you're telling us to be quiet. Why?What are you afraid of?

Re:AWS down again? by pwinnski on Friday December 10, 2021 @10:26AM (#29510953)
> The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did. Go read a few RFCs and try to imagine a scrum team today putting as much thought into an up front design.

This is a staggering bit of revisionism. As someone who was around at the time, I remember that many RFCs were written based on already-working code. They had some advantages: nobody much cared what they were doing, so they didn't have to answer to multiple levels of management, and they clearly gave almost no thought to security from bad actors, but if you think there aren't people today--including at those very cloud providers you disdain--doing work at least as well-thought-out as those early pioneers, you haven't been paying any attention at all.

I'll stay off your lawn, but maybe take off those rose-colored glasses and stop pretending the past was rosy.

Re:AWS down again? by julianlam on Friday December 10, 2021 @09:50AM (#29510490)
That's an interesting position to take, because I imagine if a team of developers decided to invent the internet in a vacuum today, it'd be a hell of a lot more secure than the "let's hope nobody uses this protocol maliciously" attitude prevalent in the early days of the internet.

Not that that is a bad thing, but just something to think about.

Re:AWS down again? by tres on Friday December 10, 2021 @10:08AM (#29510729)
It wasn't an attitude of naive hope that there wouldn't be nefarious actors leveraging the protocol; there was a different kind of people using the Internet.

There's no need to design security in the system when you literally know everyone who is using it. And everyone who was using it had the same goals in mind.

So, I don't disagree with the sentiment -- people today would probably do it a little bit differently; however, I do disagree with the expression -- people designing these protocols weren't naive. They were trustful because they had to be.

In the early days of building something new, nothing works without trust; not the Internet... not Bitcoin... not a nascent venture... nothing.

Re:AWS down again? by int0x2e on Friday December 10, 2021 @12:04PM (#29512253)
While I don't disagree, if people at the time had assumed that everyone on the network could be trusted (forever), why design the IPv4 address space to make room for 4 billion devices? Why support so many ports and concurrent connections? The two assumptions don't quite match up.
Re:AWS down again? by unionpivo on Sunday December 12, 2021 @05:09PM (#29534163)
> 4 billion devices? Why support so many ports and concurrent connections? The two assumptions don't quite match up.

Because when TCP and it's predecessors were invented there were only a few computers in entire world. In initial ARPAnet there were only 4 hosts (In September 1973 there were 42 computers connected to 36 nodes)

But each computer had many users. That's why there were so many ports, because the thinking was there will be big computers with many users each running their own internet connected clients and servers.

That was true even in the beginning of 1990's, when I want to high school, I had access to Unix shared between 2000+ people.

Re:AWS down again? by registeredcorn on Friday December 10, 2021 @12:10PM (#29512367)
To expand on what's being referenced here, consider the following: video game speedruns.

Throughout the 80s, 90s, and early-to-mid 2000s, there was a certain level of trust in the claims people made about PBs (Personal Bests) and WRs (World Record/Ranking). There was no practical way to record, host, or especially upload literal hours of footage (VHS footage) of a run you did. Even if you did somehow achieve all of the above, it would be a grainy, low quality video which is hard to see, maybe with a stopwatch nearby so people can verify your claim. People would be watching this through RealPlayer, if they could watch it at all!

So what do you do in such a situation where people have no practical or easy means to verify claims? You build credibility off of how active you are with other members of the community. You post and comment on forums about what strategies you're trying, what difficulties you're dealing with, and what new information you might have uncovered through trial and error. You don't prove your work, you prove your worth. Your standing is evidence of your claim.

To me, this is a great example of "personality-credit" communities that's existed online; Usenet and BBS aside. The mentality has largely faded away with improvements to bandwidth and services like Twitch and YouTube, but considering the technological challenges of what someone in say, 1993 would be dealing with in trying to prove they just set a new record can really give a glimpse into what things used to be like.

Re:AWS down again? by matt123456789 on Friday December 10, 2021 @10:21AM (#29510895)
People do think about it. RFCs are still being written and revised by people daily. IETF, W3C, and others are publishing new standards like QUIC. We understand the risks a lot better now than when those original RFCs were written because the world has changed a lot since then, and in no small part because of them.
Re:AWS down again? by hnlmorg on Friday December 10, 2021 @10:03AM (#29510660)
In the early days of the internet it was a closed network of academic and government properties. Nobody at that time would have guess it would grow into even the 80s style internet, let alone what we have now.
Re:AWS down again? by otabdeveloper4 on Friday December 10, 2021 @09:54AM (#29510555)
"Secure" as in "security by closed source and obscurity, because, hey come on, we need to upsell you on 'enterprise' features"?

Yes, of course. Not really any different from the Internet we have today, though.

Re:AWS down again? by PragmaticPulp on Friday December 10, 2021 @09:54AM (#29510552)
> we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead.

New cloud products are targeted at new companies and services.

If a company has already invested all of the R&D into building self-hosting and they've got it running properly with well-defined and measurable economics, it doesn't really make sense to upend it all and rebuild in the cloud.

But for new services, embracing hosted platforms is a proven accelerant for development. Skip past the solved problems like hosting, get straight to work on the interesting problems your business is trying to solve.

> The current generation couldn't invent the internet. You know why, cos they would never have the patience to spec it out like the old timers did.

Oh please. This is just "back in my day" ranting about "kids these days" and how you think one generation is superior to the other. Give it up.

Re:AWS down again? by jrochkind1 on Friday December 10, 2021 @10:13AM (#29510794)
Hm, my understanding/memory of the history of internet technologies has a lot more "let's try this and see if it works" than your comment about the role of specing it out and up-front design in the internet would suggest. I think there was actually quite a bit of people in labs just doing things, coordinating informally with people they knew personally in other labs.

Yes, there was also, at various points and times, specs and big-picture thinking, sure.

Re:AWS down again? by nonameiguess on Friday December 10, 2021 @09:51AM (#29510505)
None of that stuff is legacy. It's just centralized. Economies of scale. Go work for an infrastructure provider, the same way recruiters largely work for recruiting firms instead of all shops having their own in-house.

The fact that the people who could invent the Internet mostly work for a few giants doesn't mean they no longer exist in the current generation.

Re:AWS down again? by Spivak on Friday December 10, 2021 @10:03AM (#29510673)
(Assumedly) much much younger person here who has spent the majority of her career on prem, has lifted multiple shops to AWS, and right now works full time on an all AWS stack.

Your experience with UNIX isn't worthless, but it's worthless to anyone who's working "further up the stack" than you. If you feel like your skills are degrading then you need to find a job somewhere that's actually building infra that shops will build on top of. Your skills aren't day-to-day anymore and that's a good thing. Your generation made all that junk turnkey in the way that you probably think about dealing with Ethernet frames. Taking my first AWS job literally obsoleted all my system's knowledge -- super humbling experience -- it wasn't completely worthless but none of the problems I've spent years working out solutions even existed anymore.

You're confusing people building products with people building infrastructure -- the devops role makes this messy because you're usually "using" infra tools like a dev rather than building them. If you're working on foundational elements then literally nothing has changed. If you're shipping products then absolutely retool around a cloud provider, the infra isn't your secret sauce and if you have to move back on prem because of cost it will be good problems to have.

I mean this completely sincerely, take a job at a company that's providing hosting/services to people. All the old timers with deep deep systems knowledge are gods.

Re:AWS down again? by _hyn3 on Friday December 10, 2021 @10:33AM (#29511058)
> You're confusing people building products with people building infrastructure

Even though I'm on the side of the "UNIX graybeards" here, this is a super-great point. We do need to recognize that it is a great time to be building networked applications, precisely because the younger ones these days don't need to understand TCP/IP or anything else related to infrastructure.

I confess that I got caught up in the hype as well and built quite a few "multi-AZ" apps that I thought would help me get to five nines that much faster. (for non-cloud folks, that's 99.999% availability, which was something to pursue before the cloud.)

Of course, when those abstractions break, those same younger ones are completely helpless, and my single-server apps have been running non-stop for years at traditional providers except for a few minutes for reboots following updates. I've never had a multi-hour outage, especially one that's completely out of my control where I can only point a finger at AWS and say, "sorry, it's not my fault."

Re:AWS down again? by tuldia on Friday December 10, 2021 @09:53AM (#29510542)
This!

Not to mention the amount of garbage in the cloud, the constant learned helplessness that we have to endure even knowing that the situation could have been avoided or even mitigated/solved if the access to the box was possible.

The status-quo of the cloud is uninspiring to say the least...

Re:AWS down again? by iso1631 on Friday December 10, 2021 @10:26AM (#29510949)
I use "cloud" services, but I ensure that my systems continue even if they fail. If AWS is down, maybe I can't do some analysis on historic logs until it comes back, that's a known failure

I look at the components of a system and think "what happens if this is turned off // breaks in an unusual way // goes slow", and ensure that the predicted effects are known and acceptable based on the likelihood of failure.

That's the same whether it's an AWS managed DNS service, storage bucket, or a raspberry pi on my desk. As a systems engineer I know what that component does, what happens when it doesn't "do", and ensure the business knows how to work around it when it breaks.

If your business can't cope with an AWS outage (even if it's not as efficient) then you've got problems.

Plan for failure, and it doesn't take you by surprise.

Re:AWS down again? by hellojesus on Friday December 10, 2021 @03:16PM (#29514954)
This reminds me of something a volunteer at our campus observatory told me once while I was in college. I don't recall what we were discussing, but their comment stands out to me today: "Don't engineer things to work. Engineer things not to fail."

Often at work I see code implementations that "work" in that they usually work but can fail. I'm not a great engineer by any means (I'm actually an economist that stumbled into code), but I believe that one of the reasons I've been able to gain a good reputation at work is specifically because I design with that principle in mind.

Re:AWS down again? by strken on Friday December 10, 2021 @10:10AM (#29510756)
This conflates two very different things: cut-throat KPI-driven development practices at companies like Amazon, and the entire current generation of software developers. You're taking problems caused by the environment and attributing them to a moral deficiency in the people, which is neither fair nor helpful.
Re:AWS down again? by SQueeeeeL on Friday December 10, 2021 @10:39AM (#29511129)
Environment (in general) is what determines skill set, not the other way around. If the interview process makes you focus on leetcode skills, and your manager focuses on LOCs produced and hitting story milestones and less on spending time integrating with the team and learning about legacy codebase; it makes sense those who came out of this environment would be less prepared to tackle certain kinds of problems.
Re:AWS down again? by organic_popcorn on Friday December 10, 2021 @10:13AM (#29510810)
> The experience hasn't been great. It may simply be sour 'grapes' because, after all the expertise a whole generation has built up learning UNIX, and all the internet protocols (DNS, ARP, Email, reading RFCs, networking, routing) we get told that all that old stuff is just 'legacy,' and that we should retool around amazon's proprietary services instead.

Losing skills you worked on for years is just part of this space. We are continually building on new abstractions so that we can focus on building solutions.

This really feels like a rant of kids these days. SCRUM doesn’t mean you can’t do upfront design.

Re:AWS down again? by snovv_crash on Friday December 10, 2021 @10:23AM (#29510921)
The entirety of StackOverflow runs on something like 4 machines. Abstraction layers are expensive, and having to learn scaling methodologies unnecessarily when a better choice of upfront technology would render it unnecessary is very un-agile.
Re:AWS down again? by oxfordmale on Friday December 10, 2021 @12:46PM (#29512974)
StackOverlow doesn't run on just 4 machines. Even in 2016 it required significant hardware:

4 Microsoft SQL Servers (new hardware for 2 of them) 11 IIS Web Servers (new hardware) 2 Redis Servers (new hardware) 3 Tag Engine servers (new hardware for 2 of the 3) 3 Elasticsearch servers (same) 4 HAProxy Load Balancers (added 2 to support CloudFlare) 2 Networks (each a Nexus 5596 Core + 2232TM Fabric Extenders, upgraded to 10Gbps everywhere) 2 Fortinet 800C Firewalls (replaced Cisco 5525-X ASAs) 2 Cisco ASR-1001 Routers (replaced Cisco 3945 Routers) 2 Cisco ASR-1001-x Routers (new!)S

Re:AWS down again? by snovv_crash on Tuesday December 14, 2021 @01:07PM (#29554996)
Compare this to the many companies spending 6+ figures on AWS, and ask which has more traffic.
Re:AWS down again? by technobabbler on Friday December 10, 2021 @10:49AM (#29511262)
IMHO these are entirely different skillsets, and a division-of-labor question rather than some sort of insurmountable generation gap. It's not like infrastructural know-how isn't relevant anymore; they just became CDN engineers or DevOps or senior scaling or reliability engineers. Their jobs are no easier than before, especially when you have to consider network traversals across layers of virtualized/containerized services across multiple data centers owned by disparate parties and maintained by different vendors.

Virtualization aside, we haven't abandoned basic infrastructure, but centralized it in the hands of a few huge, expert providers. IMO this is a good thing, and was both necessary and natural as the Web grew to offer more and more opportunities to more new professionals. In detaching HTML from HTTP from ARP, etc. we gave rise to entire new professions like full-time UX (which arguably the Old Guard was never good at beyond a small audience of academics and engineers), or various flavors of front-end developer, or serverless ecosystems.

The Web and associated technologies advanced so quickly it was impractical for a single IT or network department to know all of it anymore, and some of the newer webapps wouldn't have been possible if that same team or company had to also manage all of their own basic infrastructure like it was the 90s still.

Now you can be a front-end only shop, or a UX consultant, or a network engineer who never has to touch HTML, or, or, or... maybe big enterprises always had and could always have all of those in-house, but the division of labor has been a huge boon for small businesses and startups and nonprofits, who just don't have the same resources.

As someone who grew up configuring zmodem and running BBSes and having to (mis)configure NetBIOS all the time, I am so, so glad I never have to worry about OSI layers and such ever again. It's boring to me, and the experts at it are SO much better at it, might as well let them handle it. Especially when the cost of that outsourcing is often like <$100/mo. Well worth both the time and money... and sanity. The division of concerns lets you focus on the things you're either interested in and/or good at.

Our professionals haven't gotten worse. The stack has gotten much deeper.

Re:AWS down again? by Consultant32452 on Friday December 10, 2021 @10:01AM (#29510646)
I see this as akin to how factory work has changed with modernization. In a modern highly automated factory you need a much smaller number of highly specialized engineers to maintain the robots. In the analogy, these fewer highly specialized engineers are analogous to the ones who need to know understand TCP/IP in depth and now run AWS. The rest of the workers, now replaced by robots, can move on to different productive work.
Re:AWS down again? by phailhaus on Friday December 10, 2021 @10:40AM (#29511139)
I mean, because you're wrong? Even with the recent outages, AWS still has far higher uptime and support than anything that someone could cobble together in their own small company. That's the advantage of using cloud infrastructure financed by billions of dollars.

> The current generation couldn't invent the internet.

Come on now, this is an overdone, lame argument and I can't believe you're seriously suggesting this. Do you also lament the fact that kids these days can't bind their own books? The point of tools is to be built upon, not to sit around marveling at your own genius. If you build a good tool, the folks that come after you don't have to think about it. That's how you make progress.

Re:AWS down again? by justapassenger on Friday December 10, 2021 @10:17AM (#29510847)
> The current generation couldn't invent the internet.

As an old fart myself, this is very cheap argument. Old generation wasn’t superior. Old generation couldn’t invent transistor, how useless we are!

You always move up the stack as tech progress and that’s a good thing.

Re:AWS down again? by zdragnar on Friday December 10, 2021 @10:01AM (#29510638)
> Now that because of recent events and we have a chance to be heard

Is there anything about this particular incident that is new that contributes to your position?

Maybe we run in different tech circles, but I feel like the "on-prem vs. cloud" has been litigated fairly extensively here and elsewhere. In fact, as you said yourself:

"Many of us have been taking shots at those 'big boys' since the start of this trend."

Re:AWS down again? by tjpnz on Friday December 10, 2021 @09:56AM (#29510587)
Interesting. The infra engineer at my previous company fits your description, yet he was the one pushing for our AWS migration.
Re:AWS down again? by gtirloni on Friday December 10, 2021 @09:48AM (#29510459)
Your own lack of knowledge about how to make the cloud work properly doesn't mean it's completely useless. The "old school" knowledge is still very useful in building and troubleshooting cloud-based infrastructure. You're creating a false dichotomy.
Re:AWS down again? by gerbilly on Friday December 10, 2021 @10:00AM (#29510628)
False dichotomy, if you say so.

I haven't found it too helpful when dealing with AWS and the serverless trend, whose popularity is really just based on price and economics, not technical superiority.

Serverless turns every simple system into a distributed system with the number of failure modes now multiplied by ten.

That sounds like fun.

I do know how to use serverless for the record, I just think it's an overhyped, overpriced waste of time.

Re:AWS down again? by gtirloni on Friday December 10, 2021 @10:06AM (#29510705)
You just moved the goal post from "the cloud" to "serveless". Anyway, this discussion is full of overgeneralizations that are not useful. I'm abstaining from further comments.
Re:AWS down again? by gerbilly on Friday December 10, 2021 @10:14AM (#29510814)
I'm not moving anything, just augmenting my point. Reread my comment and do s/serverless/cloud/ if you want, I still stand behind it either way.
[dead]
Re:AWS down again? by gtirloni on Friday December 10, 2021 @09:56AM (#29510578)
You're just trolling. "The cloud" doesn't exist as a single entity. I'm sure people in other various AWS regions, all GCP, all Azure, all DigitalOcean is working just fine.
Re:AWS down again? by technobabbler on Friday December 10, 2021 @09:40AM (#29510363)
Y'know, debates about cloud vs in-house uptime aside, there's one thing I'm really grateful to the cloud vendors for: making downtime somebody else's problem.

Rather than a late-night panicked run to the smoking server, now we can just shrug and wait a few hours and it fixes itself. Most websites aren't that critical, and it's nice not having to lose sleep over devops issues.

Re:AWS down again? by christophilus on Friday December 10, 2021 @09:46AM (#29510446)
I mostly agree, but this is a double-edged sword. I recently had a 6 hour outage due to lots and lots of waiting on a fix from our managed database provider. Had we been running the DB ourselves, I would have had the permissions to monitor the DB and caught the issue early (it was a pretty trivial thing to monitor). If we had gone down, I would have been able to run my restore within an hour, rather than 6. And lastly, self-hosted performance was much better than it's been with a managed DB.
Re:AWS down again? by londons_explore on Friday December 10, 2021 @09:48AM (#29510473)
I find most cloud users still have to panic at 3am when there is a cloud outage because usually there are knobs to tweak in their application to mitigate the outage. For example, do a region failover, turn off some feature that depends on dynamodb, or push an emergency release to make the application server handle spurious 403 errors from an Amazon backend.

Since Amazon failure modes are so varied, it's impossible to make the above tweaks fully automated. There'll always be a new way the cloud can misbehave, and often your service can at least maintain partial service by being nimble enough.

Re:AWS down again? by technobabbler on Friday December 10, 2021 @10:38AM (#29511120)
If you're going closer-to-the-metal cloud, then yeah. I think that's part of the allure of newer, more abstracted cloud hosts like Vercel, Netlify, Gatsby, etc.

They take care of all the infrastructure for ya, and if something breaks, you just twiddle your thumbs while they twiddle the dials.

Of course this sort of laziness is not a valid approach if you're a huge enterprise or running some live-saving project that needs seventy nines of uptime, but for non-critical websites, it's a great sigh of relief...

Re:AWS down again? by ksec on Friday December 10, 2021 @09:57AM (#29510599)
>making downtime somebody else's problem.

It is even more, making downtime's responsibility somebody else's problem. You can now simply point to AWS is down, AWS is slow, AWS is causing error, and there is nothing we can do about it :). And as long as management knows everyone is having the same problem they are perfectly fine with it.

Re:AWS down again? by SahAssar on Friday December 10, 2021 @12:40PM (#29512874)
Do people not take responsibility for their choice of vendors? If boeing crashed a bunch of planes and said "Not our fault, our subcontractor made shitty software" would we accept it? If a startup leaked all their customers data would we accept "Blame this other company, we just gave them all our data for analysis" as an excuse?

If you accept that then it means that we need to investigate and scrutinize the whole supply chain of any product before trusting it and it would also mean that any company switching even the smallest part of their supply chain would require us to reevaluate their product.

When we buy things (either products, services or software) we consider the company selling it to be responsible for the whole and deferring responsibility to a vendor is basically just saying that you were either promising more than you knew could deliver or that you did not do a good job picking vendors.

Re:AWS down again? by greedo on Friday December 10, 2021 @02:31PM (#29514400)
It used to be "No one got fired for buying IBM." Now it's AWS. Many CTOs are very risk averse, and having someone to blame is incredibly valuable to them. My company has a large multi-region set of datacenters, but we're using AWS for a lot of new projects, despite being able to deliver similar services with better performance and cost, on-prem.

Some of the reasons are no one wants to be seen using old tech; stuck being the last COBOL dev. So there's a big incentive for developers and their managers to be using the latest tech; it looks good on the devs resume and grants them job security in the field, and it makes their managers look like they're sharp and on top of new technology.

Also, recruiting people familiar with AWS services is often easier than finding someone versed in on-prem tech. And for companies in less than desirable locations, this helps remove a staffing issue since it can all be done remotely.

And when something goes wrong on-prem, the CTO is expected by the rest of the C-suite to fix the issue. When it's a reputable 3rd party, he can absolve both himself and his team of responsibility.

Re:AWS down again? by SahAssar on Friday December 10, 2021 @03:03PM (#29514810)
> he can absolve both himself and his team of responsibility.

Everything up until this sentence makes sense in a twisted sort of way, but that's the part I don't get. When did we absolve companies from their choice of vendors? If I buy a service and that service does not work then the company selling that service does not get to blame their vendors, that they chose, just like they don't get to blame individual devs that they hired.

Re:AWS down again? by greedo on Friday December 10, 2021 @04:03PM (#29515431)
The company trusts the CTO; the CTO trusts AWS, and if AWS goes down, he can deflect responsibility onto AWS. The other C-Suite officers are largely ignorant about tech (that's why they have a CTO), and nod their heads.
Re:AWS down again? by ksec on Friday December 10, 2021 @03:21PM (#29515006)
>When did we absolve companies from their choice of vendors?

>just like they don't get to blame individual devs that they hired.

When the vendors become a symbol, a public utility status or practically the only choice. And this isn't a Tech company thing. It can be applied to all other industry. You can fire and hire another Dev. If you are looking at cloud vendor, you only have AWS, Azure and GCP to choose. For lots of reason the latter two may not be an option due to competition or features. And AWS is the best you can get, are you sure the Dev they have are the best they can hire?

Re:AWS down again? by SahAssar on Friday December 10, 2021 @03:53PM (#29515332)
> a symbol, a public utility status or practically the only choice

The first, I can agree with, but the latter two are definitely not the case. If they were a public utility they would be regulated as one and if they were they only choice there would not be companies online during these outages. The site you are writing this on is an example, an online service with a pretty large (although not massive) following that is online enough to be the place where all the people who work at companies that depend on cloud services come to discuss their outages.

Re:AWS down again? by ksec on Saturday December 11, 2021 @02:02AM (#29519100)
>Public Utility

It is not whether they are one as in business or one as in politics that has to be regulated. They are utility like as people treated it as one and it behaves as one. Many things goes down with it with AWS just like many things goes down with the grid.

HN is comparatively tiny and simple. Doesn't requires any of the features on AWS. And has no reliability requirements like SaaS. HN doesn't even use CDN. Even though I am not a cloud advocate for 99% of cases it makes zero sense to build a SaaS with your own Server.

Re:AWS down again? by technobabbler on Friday December 10, 2021 @04:26PM (#29515640)
Life-critical industries like airlines, cars, healthcare, etc. are probably the exception?

If a restaurant ran out of some menu item for a few days because a supplier didn't have a good harvest, eh, it happens. If a bookstore doesn't stock a certain book, no big deal. Grocery store self-checkout broken yet again? Oh well. Band got too drunk and can't perform? Reschedule it. No snow at the slopes? Climate change.

In the real world, shit happens, and most of it isn't critical.

If you offer a 99.999% guarantee and your customers pay for it, well, you better deliver -- subcontractors or not. But many businesses don't need to do that and a few hours of downtime a year is just a minor inconvenience. To go from "a few days of downtime" to "a few hours" is really easy; just about any provider can offer that. To go from "a few hours of downtime" to "a few seconds" is a huge investment and not worth it to many folks.

Re:AWS down again? by tuldia on Friday December 10, 2021 @10:05AM (#29510690)
> You can now simply point to AWS is down

Heck, if that was even possible... "everything is green" dashboards... :-)

Re:AWS down again? by technobabbler on Friday December 10, 2021 @10:36AM (#29511086)
Obviously, the answer is to pay for a cloud dashboard-of-dashboards that show orange lights when AWS's is falsely green.

"We need this new fancy dashboard to monitor our other dashboards in one place, more accurately!"

Re:AWS down again? by emptybottle on Friday December 10, 2021 @10:11AM (#29510766)
It's not "weasels" scampering out of the wood work.

This is a community of highly technical users expressing legitimate frustration and concern that the biggest player in cloud hosting is

  a) having a service impacting event, and
  b) the status page says everything is up
That's why people talk about the beneits of alternate hosting arrangements. If the status page said what was happening, there would be no need to ask "is AWS down?" the status would be clear.
Re:AWS down again? by ignoramous on Friday December 10, 2021 @09:49AM (#29510479)
> oh boy! Time to rethink this whole cloud thing!

The problem isn't the cloud, the problem is all of the web properties going down all at once, exacerbated by the fact that AWS charges a premium for any viable cross-region / multi-cloud architecture (given its relatively high egress fees). This is discounting the inter-dependence among AWS services themselves, some of which aren't multi-region (afaik).

Re:AWS down again? by PragmaticPulp on Friday December 10, 2021 @10:04AM (#29510680)
> I expect better of the community here. All it takes is a chance to take a cheap shot at one of the “big boys” and then all of a sudden the weasels come scampering out of the wood work.

Engineers like to build stuff. Historically, running your own infrastructure and configuring servers from the ground up were the building blocks of the internet. These new cloud services took away those building blocks and they had the nerve to charge for it.

In the real world, there are more interesting problems to solve than setting up and maintaining your own servers so it isn't really a complaint. It's actually more fun to get the hosting stuff out of the way and focus on the problem at hand.

HN comments are basically notorious for exaggerating the downsides of hosted services while overplaying the ease and benefits of DIY. A lot of the comments here are similar to the famous Dropbox comment thread where users couldn't understand why anyone would want to use Dropbox when they could simply set up a complicated self-hosted rsync contraption to sync their own files.

Re:AWS down again? by doublerabbit on Friday December 10, 2021 @10:49AM (#29511269)
Serverless is just buzzword for a container on a virtual machine on a server. The Cloud is just a buzzword for a company with a data centre selling virtual servers from other big servers.

> You’re either so new to this stuff to have no experience to remember the days before cloud

I do, and it was much more peaceful.

Everything you can do on "in the cloud" you can do in colocation. Which in the long-run is cheaper, more secure* and it's yours! Including the data.

There are caveats, network, component failure. Investment in to these and you can have a pretty king setup.

The cloud enabled magnitude of email spam, brute-forcing, botnets, security vulnerabilities and much more. Operators are lazy don't want to combat it. Providers are bias, then again you can say that about any business.

People flock to the cloud like it's the greatest thing, when all your buying in to is a expensive price-plan for a company who will happily knock you off their service if you somehow brush up the wrong way and then charge you for a closed account.

I do laugh when something goes wrong for FANG. Partly, I'm cynical and want to see the world burn, but it's also these companies exploit their userbase, their staff and the environments resources. When Facebook locked themselves out of their own offices due to the BGP issue, now that's funny.

My colocation costs are:

$5000 covers for a three-year 1Gbit 2u server racking space in two different DCs. Where I have full-control, as many services as I desire, allowed to host what I desire and where the internet space is actually mine. It may be a small cube of internet but I know it's my network, my traffic.

Cloud is whitewash for me, I won't buy in to it. It has it purposes and if your happy with it, fine. But for me Colo for life.

Re:AWS down again? by gerbilly on Friday December 10, 2021 @11:17AM (#29511627)
Preach it brother!

Also, you should consider renting out space on your cube.

You could even design APIs around the internet protocols so they can create their own DNS records, mailboxes. Wait...

Re:AWS down again? by hintymad on Friday December 10, 2021 @02:22PM (#29514287)
I'd like to remind everyone about Uber's experience: no EC2-like functionality until at least 2018, probably even now. Teams would negotiate with CTO for more machines. Uber's container-based solution didn't support persistent volumes for years. Uber's distributed database was based on friendfeed's design and was notoriously harder to use than DynamoDB or Cassandra. Uber's engineers couldn't provision Cassandra instances via API. They had to fill in a 10-pager to justify their use cases. Uber's on-rack router broke back in 2017 and the networking team didn't know about it because their dashboard was not properly set up. Uber tried but failed to build anything even closer to S3. Uber's HDFS cluster was grossly inefficient and expensive. That is, Uber's productivity sucked because they didn't have the out-of-box flexibility offered by cloud.

Now, how many people think they can do better than Uber?

Re:AWS down again? by greedo on Friday December 10, 2021 @02:35PM (#29514452)
Most of those issues sound like management/process issues, not technical issues. Our staff can't provision AWS instances without extensive review...
Re:AWS down again? by unethical_ban on Friday December 10, 2021 @10:13AM (#29510795)
You're correct that the knee jerk reactions are a bit over the top, but what is so interesting is how little the benefits of cloud are discussed here on HN. People who haven't used cloud native services think it's just EC2 and someone else's computer. But the idea of compute and data all being controlled through a single set of APIs, controlled by the same security syntaxes, is pretty incredible. Of course I'm a fan of the idea of on-prem alternatives, but the unified architecture is nice.
Re:AWS down again? by dudeinjapan on Friday December 10, 2021 @07:15PM (#29517111)
Disagree on two counts.

1) Like any utility, the cloud is predicated on the assumption that it never goes down. If AWS itself goes down, it causes knock-on effects more most mid-sized (i.e. 1 region, multi-AZ) services on their platform, so we can't have it.

2) If we're going to have to accept downtime (1) then yes, it IS time we rethink the cloud--either how we can achieve no downtime, or how we achieve fault-tolerance in some other way.

Re:AWS down again? by stjohnswarts on Friday December 10, 2021 @11:32AM (#29511827)
There is nothing wrong with pointing out the warts. I wonder if you've ever been at the mercy of google/aws and their lack of customer service if you aren't representing the business $$ of a fortune 500 company?
Re:AWS down again? by cosmotic on Friday December 10, 2021 @04:14PM (#29515529)
I don't think anyone is saying they could do better than AWS at making their own infrastructure. I think the complaint is that it's a single point of failure for a huge swath of the internet.
[deleted]
Re:AWS down again? by reidfromhome on Friday December 10, 2021 @09:57AM (#29510596)
There never was an outage. It was just the marketing home page for aws.amazon.com that was down for about 30 minutes and everyone had a knee jerk reaction to “haha AWS down went down again!!” Everything that mattered works fine.
Re:AWS down again? by dimmke on Friday December 10, 2021 @09:45AM (#29510428)
Thank you for saying this.