News for hackers. Stuff that matters.
Re:Our production servers were suspended by Google Cloud by Mizoguchi on Monday March 13, 2023 @09:09AM (#35135181)
Make sure your decision to migrate to AWS is not an emotional one.

I had my own feud with AWS over a $60,000 bill and an enormous amount of hours spent by my team fighting crypto hackers after they took over one of our accounts and use very advanced scripts to create hundreds of large instances.

AWS has become very complicated to manage and at the same time the attacks to their infrastructure have increased exponentially.

If your company doesn't have a dedicated team to stay on top of AWS you'll be vulnerable to attacks like the one I experienced.

You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring.

And if you think AWS will have your back when shit hits the fan, lol good luck with that.

They will try to collect the bills while providing absolutely no support, reminding you about their Shared Responsibility Agreement.

They also expect you to fight the attacks.

I ended up spending months of work securing my services and cleaning my debt with AWS billing, to the point we almost went to court.

So again, you may have good reasons to migrate, just beware that AWS won't come free of hassle.

Re:Our production servers were suspended by Google Cloud by ragnarsson on Monday March 13, 2023 @09:14AM (#35135249)
> You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring.

What advanced knowledge are we talking about here?

Re:Our production servers were suspended by Google Cloud by djbusby on Monday March 13, 2023 @09:37AM (#35135552)
All the buried config options on AWS services to lock them down. It's a complicated platform.
Re:Our production servers were suspended by Google Cloud by jen20 on Monday March 13, 2023 @09:14AM (#35135246)
Did you get to a root cause of how the “crypto hackers” “took over” your account, or of what the “very advanced” scripts did?
Re:Our production servers were suspended by Google Cloud by mechanical_bear on Monday March 13, 2023 @09:37AM (#35135550)
Why all the quotes? It makes your response seem either sarcastic or disbelieving, either way a bit disrespectful.
Re:Our production servers were suspended by Google Cloud by nazgulsenpai on Monday March 13, 2023 @09:57AM (#35135769)
Or because he's quoting the comment in his question.
Re:Our production servers were suspended by Google Cloud by thr717272 on Monday March 13, 2023 @11:11AM (#35136777)
> Or because he's quoting the comment in his question.

On HN the usual way to quote is like I have done above.

I too react like the poster above you and would probably not quote that way except to ridicule. Hopefully there aren't many of these across my posts.

(As for an almost green account commenting on HN standards I have been here is some form or another since around 2009 I think, I just once in a while create a new account so it won't be trivially simple to doxx me.)

Re:Our production servers were suspended by Google Cloud by jen20 on Monday March 13, 2023 @11:21AM (#35136921)
It’s to quote specific words which are pertinent to the threat level implied, rather than the entire post.

Extraordinary claims require extraordinary evidence.

Re:Our production servers were suspended by Google Cloud by thr717272 on Monday March 13, 2023 @11:43AM (#35137304)
Common sense is a thing and what common sense can tell us absolutely isn't an extraordinary claim.

Edit:

By the way, what do you mean by threat level here? And FWIW, here I could have used quotes without being rude.

Also: if you read that sentence again you'll probably find that it wasn't just the use of quotes but the sentence that as a whole that made it stand out as rude.

Re:Our production servers were suspended by Google Cloud by jen20 on Monday March 13, 2023 @01:00PM (#35138623)
A baseline AWS account with MFA will not be taken over. That would require a credential leak or an IAM misconfiguration permitting access from some other account.

Suggesting it is an extraordinary claim, and would suggest an extraordinary threat to all AWS accounts. This seems very unlikely.

Re:Our production servers were suspended by Google Cloud by Mizoguchi on Monday March 13, 2023 @09:45AM (#35135638)
We had basic security like MFA set up and we never shared keys, so we don't understand until this day how they managed to take over.

The attack resembled a very aggressive, clever virus so the people behind it definitely had very advance knowledge of AWS.

They could enable regions we had disabled and recreate roles that were deleted in a matter of minutes, again MFA on, no keys.

AWS kept giving us conflicting instructions and refused to disable our account unless we stop the attack first.

We have been customers of AWS since 2014.

I believe during that time the Shared Responsibility Agreement did not exist, so AWS demanded we signed it now for them to even consider taking a look.

That's when I reached out to the Attorney General of my state who sent a letter to AWS and got things moving.

We were contacted by another team (two months later) in AWS that seemed much more knowledgeable and they gave us the scripts that eventually stopped the attack.

Our case is not that rare, if you do some research you'll fine plenty of stories like ours.

Re:Our production servers were suspended by Google Cloud by tasuki on Monday March 13, 2023 @10:22AM (#35136092)
> The attack resembled a very aggressive, clever virus so the people behind it definitely had very advance knowledge of AWS.

In what way did the attack resemble a very aggressive, clever virus?

Do you mean that it happened fast? How do you know it wasn't something simple like your credentials leaking?

Re:Our production servers were suspended by Google Cloud by jeffwask on Monday March 13, 2023 @09:45AM (#35135641)
> You simply can't count on the baseline configuration of your account and instances provided by AMS, you need advanced knowledge on how to secure your resources beyond basic stuff like MFA and monitoring.

When could you? when AWS was security by obscurity a decade ago.

When you create an account, one of the first things AWS has you acknowledge is the Shared Responsibility Model.

AWS has always given you the rope to hang yourself.

Re:Our production servers were suspended by Google Cloud by supriyo-biswas on Monday March 13, 2023 @09:55AM (#35135739)
I find the author's stance of "let's move to the competition" hilarious for this reason.

I was trying some VPN services at the time, and logged in to an AWS account with the VPN turned on, after which AWS suspended resources in the account. This took a two days to review and fix, though the only saving grace was that I did not have a paid offering running there.

When pressed as to why they'd flag my account, they'd would evade the question as to why my account was suspended, though I'm pretty sure it's the VPN. (And before you ask, no, my account was not compromised, as I had to review 90 days' Cloudtrail logs before AWS would budge on the issue).

Re:Our production servers were suspended by Google Cloud by tasuki on Monday March 13, 2023 @10:17AM (#35136009)
> very advanced scripts to create hundreds of large instances.

What was very advanced about those scripts?

As an outsider, I'd think that perhaps a very simple script might be enough to create hundreds of large instances.

Re:Our production servers were suspended by Google Cloud by Mizoguchi on Monday March 13, 2023 @10:49AM (#35136436)
Sure, but that was only one part of the attack and not the clever one. The ability to bypass security, even after following AWS support instructions is what made it so difficult to fight, to the point that only after AWS security team took over the case it did not stop. The problem is AWS security doesn't interact with customers directly (as far as I know), you have to go through tech support and these guys just don't have that level of knowledge.
Re:Our production servers were suspended by Google Cloud by philihp on Monday March 13, 2023 @10:20AM (#35136063)
AWS understands lifetime value of the user, not just the account. I've had mistakes ON MY PERSONAL ACCOUNT that were in no way connected to my work. Times when I was just experimenting. They refunded it in full. I've also heard from STUDENTS who may not ever get into infra work that they're still reasonable and lenient.

You would be absolute mad to use any other platform. AWS gets it right on the customer service front. Amazon, perhaps deeply less so... but AWS? amazing.

Re:Our production servers were suspended by Google Cloud by Mizoguchi on Monday March 13, 2023 @11:05AM (#35136699)
It was not our experience.

Tech support was always friendly, polite and apologetic, but could not provide help.

The attack was way beyond their level of skill and they insisted in providing instructions that were completely helpless, all manual actions to fight a full scale extremely vicious automated attack (change password, delete instances, disable regions, delete roles, etc).

AWS security got it done in one hour, but it took months to get to them.

Re:Our production servers were suspended by Google Cloud by rurp on Monday March 13, 2023 @03:19PM (#35141014)
A friend of mine had a similar experience when he was just learning. He accidentally racked up a large bill (for a student), contacted support, and they forgave the charges.

That's not to say that Amazon is being altruistic here, I think it just makes good business sense. Foregoing a few thousand dollars for even a tiny chance that a student's future self decides to use AWS for a multi-million spend project down the line is worth it.

Re:Our production servers were suspended by Google Cloud by kjs3 on Monday March 13, 2023 @11:45AM (#35137333)
If your company doesn't have a dedicated team to stay on top of AWS you'll be vulnerable to attacks like the one I experienced.

This has been my experience, 5+ years into our 'cloud journey'. For all those people we didn't have to have on board because cloud magically made them redundant, we've had to add almost as many people to ride herd on our relationship with the cloud providers.

TANSTAAFL

Re:Our production servers were suspended by Google Cloud by b1c1jones on Monday March 13, 2023 @12:36PM (#35138158)
I had a similar experience with AWS. They warned us of a crypto hijacking of one of our servers. They weren't any help on internally battling it. They were hands off on anything within the server, and didn't have any advice on how to address it. I had to manually battle it. Their backup and device swapping services were pretty handy though.