News for hackers. Stuff that matters.
Re:In Digital Ocean, S3-like space keys can access all your ... by snowstormsun on Sunday October 01, 2023 @07:10AM (#37724791)
And DO consistently ignores requests to add this. It prevents me from seriously using Spaces and I think many others, too.
Re:In Digital Ocean, S3-like space keys can access all your ... by sneak on Sunday October 01, 2023 @07:13AM (#37724803)
DO has had a terrible attitude to security for as long as they've been around. I reported a major data leak vulnerability to them and they told me it was operating as intended, so I published it, and then they accused me of irresponsible disclosure, while simultaneously claiming that there was no security issue. This was a dozen years ago and the founders were involved in the response.

They are at the top of the list of companies with which I will never do business.

Re:In Digital Ocean, S3-like space keys can access all your ... by k8sToGo on Sunday October 01, 2023 @07:27AM (#37724891)
Reminds me of a hosting company in my country. When you use their spaces thing, they generate a password for you (not being able to set your own is already a red flag for me, but ok). It was "hidden" in the UI to look secure. However, the password itself was passed as an argument that was part of the URL. So everyone who logs access urls could see the password.

I reported it, they fixed it quickly, but never acknowledged that it was a "security" issue.

Re:In Digital Ocean, S3-like space keys can access all your ... by Operyl on Sunday October 01, 2023 @07:27AM (#37724888)
Don’t forget the time they issued the same sshd keys to every instance because they didn’t scrub them from the images.
Re:In Digital Ocean, S3-like space keys can access all your ... by mdekkers on Sunday October 01, 2023 @07:15AM (#37724818)
Could you provide more details?
Re:In Digital Ocean, S3-like space keys can access all your ... by sneak on Sunday October 01, 2023 @07:20AM (#37724850)
On delete of VMs, the next user to be allocated disk would be able to read your data. There was a "scrub" feature that would bill you longer/extra to zeroize your data and not give it to the next customer. Apparently they hadn't heard of TRIM or thin provisioning.

https://github.com/fog/fog/issues/2525

https://news.ycombinator.com/item?id=6983097

https://news.ycombinator.com/item?id=6983520

Re:In Digital Ocean, S3-like space keys can access all your ... by belter on Sunday October 01, 2023 @07:53AM (#37725047)
Some of the comments by DO on that GitHub thread, are really surprising, and disappointing to say the least...
Re:In Digital Ocean, S3-like space keys can access all your ... by neom on Sunday October 01, 2023 @08:55AM (#37725483)
I wrote those comments, fog wasn't using the API as he would have liked, and then this guy starts a bunch of trouble on HN stating that there was a vun on DigitalOcean, and conflated someone using the API incorrectly with us having a security vun. Two things I was trying to (admittedly not very well) say was: if DO did have a security issue, posting it on HN ("DigitalOcean leaks customer data between VMs" huh?) instead of emailing us about it isn't a very responsible way to disclose it and, there isn't actually a security issue, the fog app wasn't passing the scrub flag. In retrospect, I should have been less of a jerk, but I still think sneak was acting in bad faith in how he went about the whole thing. 10 years later and I still remember how pissed off I was that day, hah.
Re:In Digital Ocean, S3-like space keys can access all your ... by rsynnott on Sunday October 01, 2023 @09:42AM (#37725867)
> there isn't actually a security issue, the fog app wasn't passing the scrub flag

At a certain point a default behaviour can be so bad, and so clearly not what a user would expect or want, that it constitutes a security issue. I would think this _more_ than qualifies.

Re:In Digital Ocean, S3-like space keys can access all your ... by neom on Sunday October 01, 2023 @10:19AM (#37726176)
You're right. I hadn't really thought about it that way. My response at the time actually came from confusion on my part (that is my own fault I suppose). I got woken up at 4am on a Sunday by a flood of calls saying there is a post on the top of HN about DO leaking data between VMs. I got out of bed to see if I needed to wake up the engineering team, checked the support system and the disclosure emails because my first thought was shit someone has pwned the LXC/escaped the jails or something. Then when I read the github, I thought the scrub flag wasn't being respected when it was passed. That was the type of security issue I had originally taken it to mean, both I would have woken up engineering for. It was a someone is in the hypervisor type of security issue I thought we were dealing with. Once I realized it was fog didn't pass the scrub flag, I wish I'd just gone back to bed and dealt with it on Monday instead of trying to do it in real time, that was stupid of me. I also accept your point that an unexpected behaviour can in itself be a security issue.
Re:In Digital Ocean, S3-like space keys can access all your ... by sneak on Sunday October 01, 2023 @09:21AM (#37725687)
Wow, this just confirms my belief.

There is precisely zero circumstance where it is ok to give private customer data to another customer.

The fact that you think it had anything to do with an API or how it is used is all anyone needs to know about it.

The idea that warning your customers of your vulnerabilities is "irresponsible" is only true if you care more about revenues than your customers' security.

Re:In Digital Ocean, S3-like space keys can access all your ... by plagiarist on Sunday October 01, 2023 @09:31AM (#37725771)
Unless I misunderstood something and one has to explicitly opt out of scrub for this to occur, I agree 100%. This is surprising. And they were mad that a customer thinks this is undesirable?

I was using DO. It is not a serious project but I think I'll be moving on nonetheless.

Re:In Digital Ocean, S3-like space keys can access all your ... by Operyl on Sunday October 01, 2023 @11:33AM (#37726871)
For what it's worth, neom has moved on from DO a while back.
Re:In Digital Ocean, S3-like space keys can access all your ... by plagiarist on Sunday October 01, 2023 @11:44AM (#37726992)
Yeah, I read a bit further and it was not scrub by default but had to be opt-in. And today is the first I heard that all API keys have ridiculous access. I'm done.

Now I just need to find a cloud hosting service that will do a low cost easy k8s cluster, preferably not AWS.

Re:In Digital Ocean, S3-like space keys can access all your ... by Operyl on Sunday October 01, 2023 @11:52AM (#37727085)
Looks like OVH does managed k8s, and they're pretty solid at this point for the most part. https://us.ovhcloud.com/public-cloud/kubernetes/
Re:In Digital Ocean, S3-like space keys can access all your ... by plagiarist on Sunday October 01, 2023 @12:27PM (#37727578)
Thank you, I didn't know they did. That puts them top-of-list for me.
Re:In Digital Ocean, S3-like space keys can access all your ... by ratg13 on Sunday October 01, 2023 @09:15AM (#37725643)
The fact that you think it should be the customers’ responsibility to make sure that DO doesn’t accidentally give your private data away to unknown third parties is an absolutely minblowing take on this matter.

I’ve been a DO customer for the better part of a decade, but there is no way that I can continue to be a customer with a company that has such a blasé stance toward security and protecting customer data.

If anyone can suggest any alternatives that are not Hetzner, I would be interested.

Re:In Digital Ocean, S3-like space keys can access all your ... by neom on Sunday October 01, 2023 @09:21AM (#37725688)
Well first, I've not worked for DigitalOcean in a long time, I left in 2015. I don't really wanna re-hash something from 2013, but again: when you deleted a VM you could click the scrub button or pass the scrub flag in the API when you issued a delete call. It was very well documented and the scrub button wasn't hidden anywhere in the interface. The fog platform wasn't using the scrub flag in their app for whatever reason. As I recall, we did change it to scrub by default shortly after. From what I remember, the reason it wasn't originally default was the zero'ing process tried up a lot of resource on the box and it was hard on the drives and we were very much in startup mode at that time with limited capacity, so it was better to only zero if the customer requests it.
Re:In Digital Ocean, S3-like space keys can access all your ... by mdpye on Sunday October 01, 2023 @09:24AM (#37725711)
You still don't appear to get it though. The fact that there even exists an option (never mind a default) which does not scrub storage between use by different tenants is completely unacceptable in a multi-tenant environment.
Re:In Digital Ocean, S3-like space keys can access all your ... by ratg13 on Sunday October 01, 2023 @09:36AM (#37725824)
Imagine if your bank told you they only close the vault when a customer asks.

And they give you an excuse that it’s time consuming for them and you should respect their time.

Would you bank with them?

This is a pathetic response and you are dragging your former company through the mud by putting the incompetence that you participated in, front and center.

It’s amazing that this happened so long ago and yet you seemed to have learned nothing from it, and now want to double down on your right to leak sensitive data (because it was convenient at the time)

Re:In Digital Ocean, S3-like space keys can access all your ... by neom on Sunday October 01, 2023 @09:44AM (#37725889)
I was always very clear at that time "this is a startup" - "do not use DO for mission critical work" - "the product is not very mature yet" - we had no sales team, nothing in our marketing was gear towards business and we primarily catered to students/personal sites/etc. I'd accept your analogy except we regularly said "use AWS for important stuff, not us".
Re:In Digital Ocean, S3-like space keys can access all your ... by ratg13 on Sunday October 01, 2023 @10:56AM (#37726482)
Even startups have to follow the law.

Before the GDPR the EU had the 1995 Data Protection Directive, which you would have to comply with to have EU customers.

You can’t just decide to not follow the law because you’re a startup.

Re:In Digital Ocean, S3-like space keys can access all your ... by neom on Sunday October 01, 2023 @11:18AM (#37726731)
What law would cover this??
Re:In Digital Ocean, S3-like space keys can access all your ... by ratg13 on Sunday October 01, 2023 @12:49PM (#37727846)
I just told you in my previous comment..

In the event your google is broken:

----

https://en.wikipedia.org/wiki/Data_Protection_Directive

>the controller must implement appropriate technical and organizational measures to protect personal data against accidental or unlawful destruction or accidental loss, alteration, unauthorized disclosure or access, in particular where the processing involves the transmission of data over a network, and against all other unlawful forms of processing.

Re:In Digital Ocean, S3-like space keys can access all your ... by neom on Sunday October 01, 2023 @01:07PM (#37728028)
Re:In Digital Ocean, S3-like space keys can access all your ... by ratg13 on Sunday October 01, 2023 @01:50PM (#37728490)
Are you confused about what PII is?

Should I explain this too?

Re:In Digital Ocean, S3-like space keys can access all your ... by 15155 on Sunday October 01, 2023 @09:48AM (#37725925)
> tried (sic) up a lot of resources on the box

Do you know what else ties up a lot of resources? Civil action, negligence claims, PR disasters.

This specific GitHub issue is one of several why I will never use DO: language, official response, and attitude all matter. I'd be curious to know how much revenue DO has lost because of this one issue.

Re:In Digital Ocean, S3-like space keys can access all your ... by plagiarist on Sunday October 01, 2023 @09:27AM (#37725742)
Why on Earth would someone have to opt-in to scrubbing with a flag vs. opting out? That's surprising. Is the flag on by default and they specifically opted out before complaining about it?
Re:In Digital Ocean, S3-like space keys can access all your ... by fullspectrumdev on Sunday October 01, 2023 @01:59PM (#37728596)
The scrub flag was default false, apparently because scrubbing was slow and caused wear.
Re:In Digital Ocean, S3-like space keys can access all your ... by richbell on Sunday October 01, 2023 @09:10AM (#37725602)
I do find the claim that "there's nothing irresponsible about full immediate disclosure" to be interesting.

https://github.com/fog/fog/issues/2525#issuecomment-31336855

Re:In Digital Ocean, S3-like space keys can access all your ... by MarkSweep on Sunday October 01, 2023 @11:46AM (#37727008)
> there isn't actually a security issue

The official blog from DigitalOcean is here:

https://www.digitalocean.com/blog/transparency-regarding-dat...

While it does not call it a “security issue” directly, it details several changes to the service to prevent customer data leaking between accounts. That seems to contradict your position that there was no problem.

> 10 years later and I still remember how pissed off I was that day, hah.

I would have hoped that after 10 years you would be able to admit that letting customers read each other’s data was a mistake. The existence of a disk scrub API, the problem being improperly reported, and DO being advertised as “not for production use” are not valid excuses.

Re:In Digital Ocean, S3-like space keys can access all your ... by figassis on Sunday October 01, 2023 @04:18PM (#37730178)
Consider the multiple S3 leaks that caused AWS to change their defaults to block public access and disable ACLs. You could say that since buckets were always default private, any misconfiguration was simply user error and users complaining were doing so in bad faith. Or you could understand the implications and help your customers protect their data.
Re:In Digital Ocean, S3-like space keys can access all your ... by 15155 on Sunday October 01, 2023 @09:35AM (#37725821)
I actually came here to add this very issue: too many people have forgotten about them.

This issue is the reason why I will never use DO or recommend it to a customer.

Re:In Digital Ocean, S3-like space keys can access all your ... by fullspectrumdev on Sunday October 01, 2023 @01:37PM (#37728348)
Holy shit that’s bad.

I wonder how one could test for that “automatically” on other providers.

Re:In Digital Ocean, S3-like space keys can access all your ... by giancarlostoro on Sunday October 01, 2023 @09:14AM (#37725632)
Yeah I was unaware of this, and will no longer consider DO for anything beyond basic VPS hosting, if even that.
Re:In Digital Ocean, S3-like space keys can access all your ... by MaxBarraclough on Sunday October 01, 2023 @09:59AM (#37726013)
See also this old comment on security issues with their managed Kubernetes offering.

https://news.ycombinator.com/item?id=22490390