The first premise when using a vendor is trust. If you don't trust them then don't use them. So far, AWS has proven to be trustworthy in my opinion. From what little I've seen about their operations they seem to give a damn and you have to assume that iceberg goes deep.

If you don't trust them, don't use them. Provide evidence to your leadership of malicious intent and provide an alternative and they'll back you.

Executives make decisions based on a small amount of evidence and go with their gut from there. So far AWS has a good reputation but if they're not careful the mood could change quickly. I don't think something like this shows any malice and I have to just assume any cloud provider _can_ access all of my data but chooses not to in order to preserve their reputation which ultimately generates more revenue long term than any single piece of data.

Why would AWS risk breaking the money printer?

This seems very binary / black-and-white to me - it’s not “you either trust them or don’t”. I may trust AWS to keep the cloud running, but I may not want to trust all their stuff with access to my private data.

If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.

If having all customer’s private S3 data being accessible by all AWS support is happening due to a wrong checkbox being selected, it definitely hints at a bigger problem and erodes my trust in that vendor.

And how they respond to this incident could restore that trust, or continue to erode it even further.

> If a provider puts themselves in a position that they’re entirely unable to access my data, or it being extremely difficult, that would actually increase my trust in them.

This is never possible if you're using KMS / server side encryption or no encryption at all. Your data on these services is always visible if they were to try to read it, whether that be forging KMS requests to decrypt data or passively snapshotting VM memory for inspection. Technically this might be solved via recent datacenter CPU security improvements but there will always be flaws and 0-days that people with money and a will can use to bypass these protections.

Use rclone with encryption for S3 and consider anything else potentially visible.

Of course, that’s why I said “extremely difficult” — it’s never completely impossible. I was thinking about Apple, who made iCloud reasonably e2e encrypted: of course, they can still push malicious updates targeting specific devices, but that needs to be a deliberate action by the organization, not a single rogue employee.

It’s not entirely unlikely that AWS was hacked / socially engineered in order to get this privilege in there, perhaps because some specific s3 buckets were being targeted.

As an organization, you need to have a high enough level of protection that these things are just not possible.

So maybe this whole idea of managed IAM policies being installed in all customers’ accounts is just a fundamentally bad idea, and should just be given on a case by case basis.

I don’t trust any data hosting. Any company can be forced to monitor it or allow government a copy.

If your data leaves your network and it’s supposed to be private, encrypt it in such a way only the people who should be able to read it can read it.

While I recognize that's the ideal goal, and most appropriate, I think there may be gaps in practice. My team just discovered yesterday that a Kinesis Firehose Stream can still deliver to an S3 bucket that lacks a bucket policy and whose ACLs disable any access to it at all. That diminished my confidence a little bit that all teams are perfectly in compliance of the overall goal.

Kinesis Firehose uses an IAM role to deliver data, so delivery within the same account does not necessarily depend on permissions on the bucket. Removing s3:* permissions from that IAM role or adding an explicit deny statement to the bucket policy would stop the flow of data.

https://docs.aws.amazon.com/firehose/latest/dev/controlling-...

https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_p...

No it can’t. The role you assigned to firehose has permission to write to the bucket.

Can you post the permissions for the IAM role, we could probably help you troubleshoot. I've NEVER heard of anything even close to this ever.

I thought this was true at the engineering level, but "authorized" cross customer data sharing- where there is some debate at what's customer data vs what's platform data- at the business level seems...rampant? Just curious for more perspective on posture.

It was like this at Facebook as well.

When I was at Facebook (2018-2020) there were in fact few permission barriers (click through the screen warning you to follow the rules, and that's at the UI level not to mention pulling stuff directly from Tao through the old PHP interfaces still laying around, or just building a user context from scratch), but plenty of auditing and automated detection.

I always did wonder how good the detection systems would be at catching people who access private data directly from Tao.

I've heard stories of same day firings, but never saw it myself. Maybe it was a myth put forth to dissuade the curious.

I have had friends who worked at Facebook brag about how easy it is for them to read anyone's messenger chat.

And at Google

KTLO?

Keeping the Lights On.

LMGTFY

I did, and it's a radio station.

The acronym was like 2nd or 3rd result on ddg after the radio station.

You’re listening to KTLO, all the hits, all day . . .

Rolls off the tongue