365 day * 24 * 0.0001 is roughly 8 hours, so it already lost the 99.99% status.
If the server didnt work - the tool too measure didnt work too! Genius
February 28, 2017. S3 went down and took down a good portion of AWS and the Internet in general. For almost the entire time that it was down, the AWS status page showed green because the up/down metrics were hosted on... you guessed it... S3.
- 2008 - https://news.ycombinator.com/item?id=116445
- 2010 - https://news.ycombinator.com/item?id=1396191
- 2015 - https://news.ycombinator.com/item?id=10033172
- 2017 - https://news.ycombinator.com/item?id=13755673 (Postmortem: https://news.ycombinator.com/item?id=13775667)
Maybe they should start using real software instead of mathematicians' toy langs
Sadly while I still use that tool a couple of jobs/companies later - I no longer recommend it because it migrated to AWS a few years back.
(For now, my out-of-AWS monitoring tool is a bunch of cron jobs running on a collections of various inexpensive vpses and my and other dev's home machines.)
(No disrespect to Nagios, I'm sure a competently managed installation is capable of being way better than what I had to put up with.)
I can’t explain why Saucelabs was the most grating one, but it was. I think it’s because they routinely experienced 100% down for 1% of customers, and we were in that one percent about twice a year. <long string of swears omitted>
When your SLA holds within a joke SLA window, you know you goofed.
"Five nines, but you didn't say which nines. 89.9999...", etc.
Our only impact was some atlassian tools.
From reading the EC2 SLA I don't think this is covered. https://aws.amazon.com/compute/sla/
The reason is the SLA says "For the Instance-Level SLA, your Single EC2 Instance has no external connectivity.". Instances that were already created kept working, so this isn't covered. The SLA doesn't cover creation of new instances.
[0] Fraction is ~ 1
The refund they give you isn’t going to dent lost revenue.
The duration of the outage in relation to that uptime is (8 h / 33602 h) * 100% = 0.024%, so the uptime is 99.976%, slightly worse than 99.99%, but clearly better than 99.90%.
They used to be five nines, and people used to say that it's not worth the while to prepare for an outage. With less than four nines, the perception might shift, but likely not enough to induce a mass migration to outage-resistant designs.
I don't think anyone would quote availability as availability in every region I'm in?
While this is their most important region, there's a lot of clients that are probably unaffected if they're not in use1.
They COULD be affected even if they don't have anything there because of the AWS services relying on it. I'm just saying that most customers that are multi region should have their east region out and are just humming along.
I don't think this is true anymore. In the early days bad enough outages in us-east-1 would bring down everything because some metadata / control pane stuff was there, I remember getting affected while in other regions, but there's been many years since this has happened.
Today for example no issues. I just avoid us-east-1 and everyone else should to. It's their worst region by far in terms of reliability because they launch all the new stuff there and are always messing it up.
I do not envy anyone working on this problem today.
We were more honest, and it probably cost us at least once in not getting business.
If you as a customer ask for 5 9s per month, with service credit of 10% of at-risk fees for missing on a deal where my GM is 30%, I can just amortise that cost and bake it into my fee.
Our company decided years ago to use any region other than us-east-1.
Of course, that doesn't help with services that are 'global', which usually means us-east-1.
1. The main one: it's the cheapest region, so when people select where to run their services they pick it because "why pay more?"
2. It's the default. Many tutorials and articles online show it in the examples, many deployment and other devops tools use it as a default value.
3. Related to n.2. AI models generate cloud configs and code examples with it unless asked otherwise.
4. It's location make it Europe-friendly, too. If you have a small service and you'd like to capture European and North American audience from a single location us-east-1 is a very good choice.
5. Many Amazon features are available in that region first and then spread out to other locations.
6. It's also a region where other cloud providers and hosting companies offer their services. Often there's space available in a data center not far from AWS-running racks. In hybrid cloud scenarios where you want to connect bits of your infrastructure running on AWS and on some physical hardware by a set of dedicated fiber optic lines us-east-1 is the place to do it.
7. Yes, for AWS deployments it's an experimental location that has higher risks of downtime compared to other regions, but in practice when a sizable part of us-east-1 is down other AWS services across the world tend to go down, too (along with half of the internet). So, is it really that risky to run over there, relatively speaking?
It's the world's default hosting location, and today's outages show it.
In every SKU I've ever looked at / priced out, all of the AWS NA regions have ~equal pricing. What's cheaper specifically in us-east-1?
> Europe-friendly
Why not us-east-2?
> Many Amazon features are available in that region first and then spread out to other locations.
Well, yeah, that's why it breaks. Using not-us-east-1 is like using an LTS OS release: you don't get the newest hotness, but it's much more stable as a "build it and leave it alone" target.
> It's also a region where other cloud providers and hosting companies offer their services. Often there's space available in a data center not far from AWS-running racks.
This is a better argument, but in practice, it's very niche — 2-5ms of speed-of-light delay doesn't matter to anyone but HFT folks; anyone else can be in a DC one state away with a pre-arranged tier1-bypassing direct interconnect, and do fine. (This is why OVH is listed on https://www.cloudinfrastructuremap.com/ despite being a smaller provider: their DCs have such interconnects.)
For that matter, if you want "low-latency to North America and Europe, and high-throughput lowish-latency peering to many other providers" — why not Montreal [ca-central-1]? Quebec might sound "too far north", but from the fiber-path perspective of anywhere else in NA or Europe, it's essentially interchangeable with Virginia.
Just go to the EC2 pricing page and change from us-east-1 to us-west-1
Incentivize the best behaviors.
Or is there a perspective I don't see?
Most people (myself include) only choose it because its the cheapest. If multiple regions were the same price then there'd be less impact if one goes down.
A negligible cost difference shouldn't matter when your apps are unstable due to the region being problematic.
agreed, but a sizable cohort of people don't have the foresight or incentives for think past their nose and clicking the cheapest option.
So its on Amazon to incentivize what's best.
This is the biggest one isn't it? I thought Route 53 isn't even available on any other region.
And looked at from the perspective of an individual company, as a customer of AWS, the occasional outage is usually an acceptable part of doing business.
However, today, we’ve seen a failure that has wiped out a huge number of companies used by hundreds of millions - maybe billions - of people, and obviously a huge number of companies globally all at the same time. AWS has something like 30% of the infra market so you can imagine, and most people reading this will to some extent have experienced, the scale of disruption.
And the reality is that whilst bigger companies, like Zoom, are getting a lot of the attention here, we have no idea what other critical and/or life and death services might have been impacted. How many houses were burgled today because Ring has been down for around 8 out of the last 15 hours (at least as I measure it)?
I don’t think that’s OK, and I question the wisdom of companies choosing AWS as their default infra and hosting provider. It simply doesn’t seem to be very responsible to be in the same pond as so many others.
Were I a legislator I would be casting a somewhat baleful eye at AWS as a potentially dangerous monopoly, and see what I might be able to do to force organisations to choose from amongst a much larger pool of potential infra, and I would be doing that because these kinds of incidents will only become more serious as time goes on.
It's the same thing here. Do you think other providers are better? If people moved to other providers, things would still go down, more likely than not it would be more downtime in aggregate, just spread out so you wouldn't notice as much.
At least this way, everyone knows why it's down, our industry has developed best practices for dealing with these kinds of outages, and AWS can apply their expertise to keeping all their customers running as long as possible.
More importantly you appear to have misunderstood the scenario I’m trying to avoid, which is the precise situation we’ve seen in the past 24 hours where a very large proportion of internet services go down all at the same time precisely because they’re all using the same provider.
And then finally the usual outcome of increased competition is to improve the quality of products and services.
I am very aware of the WWII bomber story, because it’s very heavily cited in corporate circles nowadays, but I don’t see that it has anything to do with what I was talking about.
AWS is chosen because it’s an acceptable default that’s unlikely to be heavily challenged either by corporate leadership or by those on the production side because it’s good CV fodder. It’s the “nobody gets fired for buying IBM” of the early mid-21st century. That doesn’t make it the best choice though: just the easiest.
And viewed at a level above the individual organisation - or, perhaps from the view of users who were faced with failures across multiple or many products and services from diverse companies and organisations - as with today (yesterday!) we can see it’s not the best choice.
Reality is, though, that you shouldn't put all your eggs in the same basket. And it was indeed the case before the cloud. One service going down would have never had this cascade effect.
I am not even saying "build your own DC", but we barely have resiliency if we all rely on the same DC. That's just dumb.
That is the point, though: Correlated outages are worse than uncorrelated outages. If one payment provider has an outage, chose another card or another store and you can still buy your goods. If all are down, no one can shop anything[1]. If a small region has a power blackout, all surrounding regions can provide emergency support. If the whole country has a blackout, all emergency responders are bound locally.
[1] Except with cash – might be worth to keep a stash handy for such purposes.
The internet was originally intended to be decentralised. That decentralisation begets resilience.
That’s exactly the opposite of what we saw with this outage. AWS has give or take 30% of the infra market, including many nationally or globally well known companies… which meant the outage caused huge global disruption of services that many, many people and organisations use on a day to day basis.
Choosing AWS, squinted at through a somewhat particular pair of operational and financial spectacles, can often make sense. Certainly it’s a default cloud option in many orgs, and always in contention to be considered by everyone else.
But my contention is that at a higher level than individual orgs - at a societal level - that does not make sense. And it’s just not OK for government and business to be disrupted on a global scale because one provider had a problem. Hence my comment on legislators.
It is super weird to me that, apparently, that’s an unorthodox and unreasonable viewpoint.
But you’ve described it very elegantly: 99.99% (or pick the number of 9s you want) uptime with uncorrelated outages is way better than that same uptime with correlated, and particularly heavily correlated, outages.
That homogeneity is a systemic risk that we all bear, of course. It feels like systemic risks often arise that way, as an emergent result from many individual decisions each choosing a path that truly is in their own best interests.
And at this point I’m looking at the problem and thinking, “how do we do that other than by legislating?”
Because left to their own devices a concerningly large number of people across many, many organisations simply follow the herd.
In the midst of a degrading global security situation I would have thought it would be obvious why that’s a bad idea.
At this point, is there any cloud provider that doesn't have these problems? (GCP is a non-starter because a false-positive YouTube TOS violation get you locked out of GCP[1]).
[1]: https://9to5google.com/2021/02/26/stadia-port-of-terraria-ca...
It is an extremely fundamental level of incompetence at Google. One should "figure out" the viability of placing all of one's eggs in the basket of such an incompetent partner. They screwed the authentication issue up and, this is no slippery slope argument, that means they could be screwing other things up (such as being able to contact a human for support, which is what the Terraria developer also had issues with).
Of course using your personal account with a provider for both personal and commercial services, means that that personal account getting banned would lock you out of those commercial services.
This is true of literally any provider that 1. has users accounts, 2. has many individual users, and 3. has some kind of commerce that can be conducted through the platform.
(Consider: if you did all your business's commerce through your personal PayPal account, and then one day tried to sell something on eBay and the buyer filed a claim against you that made PayPal mad, now you've "lost your PayPal account." This is a thing that happens, all the time!)
And every time, in every case, the answer is the same: no one with an ounce of ops thinking should be using the same account on the same provider for both personal and commercial activity.
Every one of these providers that allows commercial use-cases on their platform, not only allows but either encourages or even mandates creating a separate account for business use.
---
In GCP's case, it's even more clear, because GCP is actually incredibly wonky to use if you're not using it with a separate, GCP-specific account — namely, a Google Workspace account, under a Google Workspace that's 1:1 bound to a GCP organization.
Here's how you're meant to use GCP for commercial purposes:
1. Buy a corporate domain name (let's call it `corp`)
2. Sign up for Google Workspace, and associate the `corp` domain name with the workspace
3. Create a user account on that domain (you@corp)
4. Create a new Google Chrome profile (I'd recommend setting a specific window color); and in that profile, sign into that Google Workspace you@corp account for sync.
5. Run through the GCP onboarding as you@corp — tying `corp` to the resulting GCP org, and making you the Owner of `corp`.
6. Create a separate Organization Billing Account for `corp`, that invoices to your legal company, not you personally. (You can still use your personal credit card if you want — though that's still jank compared to giving them a corporate credit card through a service like Float, since that card can be in the business's name.)
6. Only use this you@corp-synced Chrome profile for work. Never use it for any non-work-related activities on Google properties. (Hint: go into the Google Workspace admin dashboard and disable all Google services you don't want that user to be allowed to sign into. Yes, you can disable YouTube.) This should be easy, because you can just keep those activities confined to your personal-account-synced Chrome-profile windows.
7 (optional). If you like, forward your "work email" (the user's Google Workspace email) to your personal email, so you don't have to check it separately. (Personally, I prefer to just be signed into both. The Gmail mobile app allows for multiple logins with both unified + separate inbox views, precisely because they expect people to keep things separated like this.)
---
How do I know that this is how you're "meant to" use GCP? What do I mean by "wonky"?
GCP expects you to be able to create new, real email addresses under a domain in order to be able to create additional users in your GCP Project and assign IAM permissions to them. Because there are org-level permissions, above the project level, that are inherently associated with the org's domain. Because GCP Identity-Aware Proxy expects you to SSO through it with Google Workspace, and only lets you use personal gmail accounts as "external" entities if you let "anyone" sign in. Because user groups from an IAM [and even GKE binding] perspective are literally just Google Workspace mailing-list groups. Etc.
GCP is 99% designed around the assumption that you have a Google Workspace and are visiting the GCP dashboard while signed into a Google Workspace account owned/managed by the Google Workspace's associated org. To Google, "Google Workspace account" == "is a business"; "Gmail account" == "is not a business".
The Terraria dev couldn't get support from Google? It's because they weren't a business in Google's eyes, just a random anonymous user.
Google closed the account unilaterally? They don't do that to businesses (I promise; they send nastygrams if they detect you doing weird shit in GCP, but they don't shut anything down, because they know businesses sometimes do weird shit.) But they didn't know the Terraria dev had a business, because all they saw was a personal account.
---
But why does GCP let you even use it without a Google Workspace account?
Because there are personal, non-commercial things a person can do with GCP (either explicitly or implicitly), and GCP supports doing these few things with a personal Gmail account.
One major place this comes up is in Google Apps Script. Unlike in Microsoft Office, Google Docs, Google Sheets, Gmail, etc don't have a client-side scripting model. Instead, all user-supplied scripting-type automations inside these services execute on the backend. Which means they consume resources. Which means that, to run such scripts (esp. persistently), the user needs a "compute-resource-allocating container" attached to them — i.e. a GCP project. (This also means that the resource consumption by these automations gets billed to the user — with these bills just being eaten by the usage falling within GCP's free tier.)
---
But you might be able to see how it makes sense, from Google's perspective, for them to blindly shut down a personal account's GCP usage along with everything else going on attached to that personal Gmail account, if they violate the Google TOS.
Because a personal account's GCP usage is supposed to be for, like, scripts that periodically auto-reorganize your Gmail inbox. Or for Sheets that periodically scrape a JSON file to keep cells pre-populated with fresh data. Not for running your dang business on!
Also, Steve Jobs already wrote your comment better. You should have just stolen it. “You’re holding it wrong”.
But I'll rephrase it, if you can't read the less-charitable interpretation between the lines:
99.99% of personal users who have any GCP resources, either don't realize they have them and don't care (the Google Apps Script case — shutting off those resources is immaterial to them), or they are actively-malicious users trying to set up bunches of Google accounts to use GCP free-tier promotional resource spend to run VMs to join into botnets to DDoS people with / sell into commercial VPN layers / send spam with / etc.
Every IaaS, VPS, bare-metal host, etc. treats their individual users like shit, because the same "easy on-ramp" that marketing departments argue for means that there's no KYC process to deduplicate users by their true identity, and so nothing standing in the way of a user coming back over and over (through residential VPNs, with generated $1-limit credit cards) to make new accounts, get the promotional credits, and spend them on whatever, before vanishing into the night without paying.
Due to this, individual user accounts — ones with running resources, who may very well be businesses too stupid to register as businesses! — must be able to be terminated with extreme prejudice, with no human veto. Because a human veto is too slow compared to the speed and scale at which these attackers can operate; and so the literal only other option (under the constraint of allowing that easy on-ramp) is that the entire service gets consumed by a botnet, and the QoS for paying customers drops to zero.
It just-so-happens that Google is both an IaaS and a consumer web services company. But that just ramps up this existing incentive higher: the median YouTube-TOS-violator is some kind of comment spammer; and that user account is likely also taking advantage of GCP promotional credit to run the very bot that is posting the spam comments. If Google bans you because they think you're spamming YouTube comments, then why wouldn't they also assume that any GCP resources you're using are involved in operating a comment-spam bot? That's the pattern they've seen — millions of times by now, given that these register-an-account-and-provision-GCP-resources workflows have themselves been automated by the attackers.
And again, none of this happens if you actually go through the explicit KYC process involved in signing up for Google Workspace + signing up to GCP as a business + setting up an org Billing Account. Because then you no longer look like a random malicious attacker; you look like a business customer. And business customers get the benefit of the doubt.
> You're holding it wrong
Imagine you're a doctor in an active war zone. People could shoot at you, but they don't, because they can tell you're a doctor. But they can tell only because you dress a certain recognizable way — showing recognizable symbols of your profession and purpose there (prominent Red Cross symbol, etc.)
There is, in fact, a wrong way to demonstrate that you are a humanitarian non-combatant in a warzone. (That being: failing to even attempt to do anything to signal it until after people already have guns pointed at you.)
And, insofar as malicious actors vs hosting providers is also a kind of active war zone — one where your account could catch a stray bullet by mistake — there is in fact a wrong way to signal that you are not one of the belligerents. (That being: failing to even attempt to make yourself and your business known in detail to the hosting provider as a distinct entity to be recognized and treated as a B2B customer, until after they already close your account for a TOS violation.)
Does Google warn you about this when you sign up? No.
And PayPal having the same problem in no way identifies Google. It just means that PayPal has the same problem and they are also incompetent (and they also demonstrate their incompetence in many other ways).
Sorry
No, thank you.
Do you consider regular brick-and-mortar savings banks to be incompetent when they freeze someone's personal account for receiving business amounts of money into it? Because they all do, every last one. Because, again, they expect you to open a business account if you're going to do business; and they look at anything resembling "business transactions" happening in a personal account through the lens of fraud rather than the lens of "I just didn't realize I should open a business account."
And nobody thinks this is odd, or out-of-the-ordinary.
Do you consider municipal governments to be incompetent when they tell people that they have to get their single-family dwelling rezoned as mixed-use, before they can conduct business out of it? Or for assuming that anyone who is conducting business (having a constant stream of visitors at all hours) out of a residentially-zoned property, is likely engaging in some kind of illegal business (drug sales, prostitution, etc) rather than just being a cafe who didn't realize you can't run a cafe on residential zoning?
If so, I don't think many people would agree with you. (Most would argue that municipal governments suppress real, good businesses by not issuing the required rezoning permits, but that's a separate issue.)
There being an automatic level of hair-trigger suspicion against you on the part of powerful bureaucracies — unless and until you proactively provide those bureaucracies enough information about yourself and your activities for the bureaucracies to form a mental model of your motivations that makes your actions predictable to them — is just part of living in a society.
Heck, it's just a part of dealing with people who don't know you. Anthropologists suggest that the whole reason we developed greeting gestures like shaking hands (esp. the full version where you pull each-other in and use your other arms to pat one-another on the back) is to force both parties to prove to the other that they're not holding a readied weapon behind their backs.
---
> Are you warned about the risks in an active war one? Yes. Does Google warn you about this when you sign up? No.
As a neutral third party to a conflict, do you expect the parties in the conflict to warn you about the risks upon attempting to step into the war zone? Do you expect them to put up the equivalent of police tape saying "war zone past this point, do not cross"?
This is not what happens. There is no such tape. The first warning you get from the belligerents themselves of getting near either side's trenches in an active war zone, is running face-first into the guarded outpost/checkpoint put there to prevent flanking/supply-chain attacks. And at that point, you're already in the "having to talk yourself out of being shot" point in the flowchart.
It has always been the expectation that civilian settlements outside of the conflict zone will act of their own volition to inform you of the danger, and stop you from going anywhere near the front lines of the conflict. By word-of-mouth; by media reporting in newspapers and on the radio; by municipal governments putting up barriers preventing civilians from even heading down roads that would lead to the war zone. Heck, if a conflict just started "up the road", and you're going that way while everyone's headed back the other way, you'll almost always eventually be flagged to pull over by some kind stranger who realizes you might not know, and so wants to warn you that the only thing you'll get by going that way is shot.
---
Of course, this is all just a metaphor; the "war" between infrastructure companies and malicious actors is not the same kind of hot war with two legible "sides." (To be pedantic, it's more like the "war" between an incumbent state and a constant stream of unaffiliated domestic terrorists, such as happens during the ongoing only-partially-successful suppression of a populist revolution.)
But the metaphor holds: just like it's not a military's job to teach you that military forces will suspect that you're a spy if you approach a war zone in plainclothes; and just like it's not a bank's job to teach you that banks will suspect that you're a money launderer if you start regularly receiving $100k deposits into your personal account; and just like it's not a city government's job to teach you that they'll suspect you're running a bordello out of your home if you have people visiting your residentially-zoned property 24hrs a day... it's not Google's job to teach you that the world is full of people that try to abuse Internet infrastructure to illegal ends for profit; and that they'll suspect you're one of those people, if you just show up with your personal Google account and start doing some of the things those people do.
Rather, in all of these cases, it is the job of the people who teach you about life — parents, teachers, business mentors, etc — to explain to you the dangers of living in society. Knowing to not use your personal account for business, is as much a component of "web safety" as knowing to not give out details of your personal identity is. It's "Internet literacy", just like understanding that all news has some kind of bias due to its source is "media literacy."
I am in the middle of convincing the company I just joined to consider building on GCP instead of AWS (at the very least, not to default to AWS).
Also, who’s using email-based OTP?
I would think a lot of clients would want that.
On AWS's side, I think us-east-1 is legacy infrastructure because it was the first region, and things have to be made replicable.
For others on AWS who aren't AWS themselves: because AWS outbound data transfer is exorbitantly expensive. I'm building on AWS, and AWS's outbound data transfer costs are a primary design consideration for potential distribution/replication of services.
And yes, AWS' rates are highway robbery. If you assume $1500/mo for a 10 Gbps port from a transit provider, you're looking at $0.0005/GB with a saturated link. At a 25% utilization factor, still only $0.002/GB. AWS is almost 50 times that. And I guarantee AWS gets a far better rate for transit than list price, so their profit margin must be through the roof.
Which makes sense, but even their rates for traffic between AWS regions are still exorbitant. $0.10/GB for transfer to the rest of the Internet somewhat discourages integration of non-Amazon services (though you can still easily integrate with any service where most of your bandwidth is inbound to AWS), but their rates for bandwidth between regions are still in the $0.01-0.02/GB range, which discourages replication and cross-region services.
If their inter-region bandwidth pricing was substantially lower, it'd be much easier to build replicated, highly available services atop AWS. As it is, the current pricing encourages keeping everything within a region, which works for some kinds of services but not others.
This aligns with their “you should be in multiple AZs” sales strategy, because self-hosted and third-party services can’t replicate data between AZs without expensive bandwidth costs, while their own managed services (ElastiCache, RDS, etc) can offer replication between zones for free.
In general it seems like Europe has the most internet of anywhere - other places generally pay to connect to Europe, Europe doesn't pay to connect to them.
The other concerns could have to do with the impact of failover to the backup regions.
"Changes that I make are not always immediately visible": - "...As a service that is accessed through computers in data centers around the world, IAM uses a distributed computing model called eventual consistency. Any changes that you make in IAM (or other AWS services), including attribute-based access control (ABAC) tags, take time to become visible from all possible endpoints. Some delay results from the time it takes to send data from server to server, replication zone to replication zone, and Region to Region. IAM also uses caching to improve performance, but in some cases this can add time. The change might not be visible until the previously cached data times out...
...You must design your global applications to account for these potential delays. Ensure that they work as expected, even when a change made in one location is not instantly visible at another. Such changes include creating or updating users, groups, roles, or policies. We recommend that you do not include such IAM changes in the critical, high availability code paths of your application. Instead, make IAM changes in a separate initialization or setup routine that you run less frequently. Also, be sure to verify that the changes have been propagated before production workflows depend on them..."
https://docs.aws.amazon.com/IAM/latest/UserGuide/troubleshoo...
us-east-1 is so the government to slurp up all the data. /tin-foil hat
I'm not sure a lot of companies are really looking at the costs of multi-region resiliency and hot failovers vs being down for 6 hours every year or so and writing that check.
Our stuff is all in us-east-1, ops was a total shitshow today (mostly because many 3rd party services besides aws were down/slow), but our prod service was largely "ok", a total of <5% of customers were significantly impacted because existing instances got to keep running.
I think we got a bit lucky, but no actual SLAs were violated. I tagged the postmortem as Low impact despite the stress this caused internally.
We definitely learnt something here about both our software and our 3rd party dependencies.
Amazon says service is now just "degraded" and recovering, but searching for products on Amazon.com still does not work for me. https://health.aws.amazon.com/health/status
When this is fixed, I am very interested in seeing recorded spend for Sunday and Monday.
That number had dropped to 1,190 by 4:22 AM Pacific (7:22 AM Eastern).
However, that number is back up with a vengeance. 9,230 reports as of 9:32 AM Pacific (12:32 Eastern).
Part of that could be explained by more people making reports as the U.S. west coast awoke. But I also have a feeling that they aren't yet on top of the problem.
If it doesn’t stop, that means it has a battery backup. But you can still make life more bearable. Switch off all your breakers (you probably have a master breaker for this), then open up the alarm box and either pull the battery or - if it’s non-removable - take the box off the wall, put it in a sealed container, and put the sealed container somewhere… else. Somewhere you can’t hear it or can barely hear it until the battery runs down.
Meanwhile you can turn the power back on but make sure you’ve taped the bare ends of the alarm power cable, or otherwise electrically insulated them, until you’re able to reinstall it.
I would honestly do your box option. Stuff it in there with some pillows and leave it in the shed for a while.
I.e. lots of folks that weren't expected to work today and/or trying to round them up to work the problem.
In my experience, the teams at AWS are pretty diverse, reflecting the diversity in the area. Even if a lot of the Indian employees are taking the day off, there should be plenty of other employees to back them up. A culturally diverse employee base should mitigate against this sort of problem.
If it does turn out that the outage was prolonged due to one or two key engineers being unreachable for the holiday, that's an indictment of AWS for allowing these single points of failure to occur, not for hiring Indians.
All the schools in the area have days off for Indian Holidays since so many would be out of school otherwise.
There are 153k Amazon employees based in India according to LinkedIn.
My main beef with that team was that we worked on too many stories in parallel so information on brand new work was siloed. Everyone caught up after a bit but stuff we just or hadn’t demoed yet was spotty for coverage.
If I was up at 1 am it was because I had insomnia and figured out exactly what the problem was and it was faster to fix it than to explain. Or if I wake up really early and the problem is still not fixed.
By the way, Twilio is also down, so all those login SMS verification codes aren’t being delivered right now.
A lot of these are second order dependencies like Astronomer, Atlassian, Confluent, Snowflake, Datadog, etc... the joys of using hosted solutions to everything.
When the NAS shit the bed, we lost half of production and all our run books. And we didn’t have autoscaling yet. Wouldn’t for another 2 years.
Our group is a bunch of people that has no problem getting angry and raising voices. The whole team was so volcanically angry that it got real quiet for several days. Like everyone knew if anyone unclenched that there would be assault charges.
Rest and vest CEOs
He got a lot of impossible shit done as COO.
They do need a more product minded person though. If Jobs was still around we’d have smart jewelry by now. And the Apple Watch would be thin af.
https://health.aws.amazon.com/health/status?path=open-issues
The closest to their identification of a root cause seems to be this one:
"Oct 20 8:43 AM PDT We have narrowed down the source of the network connectivity issues that impacted AWS Services. The root cause is an underlying internal subsystem responsible for monitoring the health of our network load balancers. We are throttling requests for new EC2 instance launches to aid recovery and actively working on mitigations."
Even the acronym is fucked.
My favorite by a large margin...
https://en.wikipedia.org/wiki/List_of_military_slang_terms#F...
Not to be confused with "Foobar" which apparently originated at MIT: https://en.wikipedia.org/wiki/Foobar
TIL, an interesting footnote about "foo" there:
'During the United States v. Microsoft Corp. trial, evidence was presented that Microsoft had tried to use the Web Services Interoperability organization (WS-I) as a means to stifle competition, including e-mails in which top executives including Bill Gates and Steve Ballmer referred to the WS-I using the codename "foo".[13]'
There are documented uses of FUBAR back into the '40s.
Somewhat common. Comes from the US military in WW2.
I remember a meme years ago about Nestle. It was something like: GO ON, BOYCOT US - I BET YOU CAN’T - WE MAKE EVERYTHING.
Same meme would work for Aws today.
Not really, there are enough alternatives.
And it’s not lie there aren’t other brands of chocolate either…