News for hackers. Stuff that matters.
Re:AWS EC2 Having Issues by bashinator on Tuesday April 13, 2021 @06:19PM (#26800099)
Never run prod in us-east-1.
Re:AWS EC2 Having Issues by pg_bot on Tuesday April 13, 2021 @07:20PM (#26800820)
I would argue the opposite. Run in us-east-1 if you want to have truly reliable infrastructure.
Re:AWS EC2 Having Issues by bigiain on Tuesday April 13, 2021 @08:01PM (#26801274)
AWS cost saving pro tip: Save that 10c per minute Fault Injection Simulator charge by running in us-east-1...
Re:AWS EC2 Having Issues by busterarm on Tuesday April 13, 2021 @08:58PM (#26801722)
Agree. 10s of millions in infrastructure running in us-east-1 here. Honestly it seems to be the only reliable way to teach people about failure and the importance of reserving capacity.
Re:AWS EC2 Having Issues by meepmorp on Tuesday April 13, 2021 @06:22PM (#26800133)
It's the one with the chaos monkey built in.
Re:AWS EC2 Having Issues by luhn on Tuesday April 13, 2021 @06:32PM (#26800279)
Re:AWS EC2 Having Issues by meepmorp on Tuesday April 13, 2021 @07:39PM (#26801035)
Kind of like Donkey Kong, but he's rolling flaming barrels down the hot aisles.
Re:AWS EC2 Having Issues by pclmulqdq on Tuesday April 13, 2021 @07:06PM (#26800646)
Some of us have to. Weirdly, the bitcoin exchanges are all there.
Re:AWS EC2 Having Issues by johncena33 on Tuesday April 13, 2021 @06:49PM (#26800465)
I see this a lot. Is there any reason why this particular region will have so many issues?
Re:AWS EC2 Having Issues by TheDong on Tuesday April 13, 2021 @06:55PM (#26800528)
EC2 existed in us-east-1 only for 2 years before it was launched in a second region.

Those 2 years were already enough to pick up some legacy cruft and special cases. us-east-1 is also the first region for various other old services, like s3 and route53.

I think it's also the largest region across many axis, which naturally means it will hit various scaling issues before other regions.

Re:AWS EC2 Having Issues by notatoad on Tuesday April 13, 2021 @06:50PM (#26800490)
A big reason is it’s the largest region. More servers = more likely to have problems
Re:AWS EC2 Having Issues by ufoolme on Tuesday April 13, 2021 @06:56PM (#26800535)
Not sure if it is related but it is usually one of the first to get or be trialing new features. I believe a few global services are run out of us-east-1 as well.
Re:AWS EC2 Having Issues by guidedlight on Tuesday April 13, 2021 @06:59PM (#26800575)
This is my understanding as well. New features are rolled out in us-east-1 first, then progressively through the remainder of the regions.
Re:AWS EC2 Having Issues by WatchDog on Tuesday April 13, 2021 @07:36PM (#26801002)
I remember being bitten by an RDS update, running in ap-southeast-2.

Took out a few of our production databases, seems like Australia is the beta testing region for at least a few services.

At least in the US you have the choice of other regions.

Re:AWS EC2 Having Issues by acdha on Tuesday April 13, 2021 @06:37PM (#26800330)
That joke is getting pretty tired by now and restating it seems unlikely to be helpful for anyone who's affected by the rate-limited API calls in question.
Re:AWS EC2 Having Issues by igetspam on Tuesday April 13, 2021 @06:54PM (#26800520)
It's not a joke. The us-east-1 region is one of if not the oldest. Some of the infrastructure is aging because it's hard to phase out. Amazon has learned a lot since then and the newer regions have reaped all the benefits, where Virginia hasn't. There are components that exist in that region and don't exist in others but as a rule, I use those as remote resources and keep all my critical infra in Ohio, if I need to be near east.
Re:AWS EC2 Having Issues by acdha on Tuesday April 13, 2021 @07:36PM (#26801005)
Have you actually measured it? I’ve had production workloads in us-east-1 for over a decade and haven’t seen it be notably different than the other regions. There’ve been a handful of times where there’s been a truly regional outage (17 minutes of network connectivity in 2011) and a couple of broader service outages (S3, IAM), but it’s not like the other regions never have problems.

In general most of the time when I’ve heard people say “us-east-1 is unreliable” it’s ended up that they’re really telling you that they were in just one AZ, thought the RDS multiple AZ mode was too expensive, etc. and standard architectural guidance would have taken care of it.