News for hackers. Stuff that matters.
Re:I deleted data from production by deathanatos on Tuesday October 26, 2021 @12:09AM (#28996297)
I've had someone do, in essence, a `terraform destroy`. I don't think we ever learned if it was that, or some other command … just that it was indeed a terraform removal. The plan was not read, of course. Instant incident.

More recently, had a coworker want to run terraform plan, couldn't, because the state file is access controlled, and so he just reset the configuration locally to a blank local state, plan (which wasn't read, again), and then subsequently created a lot of duplicate resources. (Since TF was essentially starting from a blank slate.)

We had an admin interface that used a generic UI; every database table was just a generic CRUD "it's a table, here's the columns & values" to it. The mobile app was an OAuth app, just like any third-party app was. One day, someone on the QA team (why did QA have access to prod, you ask? Good question…) deleted the mobile app's OAuth client from the database. "Why did you do this?" :shrug: "Why do we even have this lever, Kronk?" I suggested at the time we should have kept the delete button, but special cased it to remove the admin privs of anyone who clicked it, since clearly they failed the test…

I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no? (I'm not sure I'd want to try that on a live OS … and … if this is the cloud, usually growing the underlying EBS or cloud disk or whatever is pretty easy to do? But I also usually try to keep any "it could grow" data on a separate partition to spare / the pain…) Though I think I'd bind-mount it back in place, not symlink it.

Re:I deleted data from production by anyfoo on Tuesday October 26, 2021 @01:47AM (#28996832)
> I can't tell why the copy in the article didn't work? You should be able to move /usr to another partition, no?

The way it reads, they didn't know how POSIX permissions work, and maybe still don't. The article says that sudo needs to be owned by uid 0, but does not mention the critical suid bit. Maybe they just recursively copied /usr without preserving permissions?

Also, the mention of sudo makes it pretty clear that they did not do this in an interactive root shell (obtainable with sudo -s/-i or su if there is a root pw). If you do this kind of shenanigans, you better have multiple open root shells in front of you, and some statically linked binaries (busybox?) to recover from the inevitable chaos of missing binaries, shared libraries, and dynamic loaders that you'll transiently have.[1] It's not impossible to perform, but requires careful planning and a good understanding.

[1] Though if /usr was indeed copied without preserving permissions, recovering would have been very tedious.

Re:I deleted data from production by blef on Tuesday October 26, 2021 @04:25AM (#28997788)
Yeah I made some shortcuts about it in the post for the sake of trying to keep it short. But the biggest issue with sudo was the suid bit you're right.

You're totally right in your message. I did not had access to root password.

I was junior and while I was trying to fix stuff I was adding more problem to the initial one that was a disk space issue.