News for hackers. Stuff that matters.
Re:Facebook Down? by viraptor on Thursday April 08, 2021 @05:46PM (#26744210)
At that size of a company there's usually blast radius restrictions and per-role permissions. I don't expect anyone has enough rights to "delete Facebook" on their own.
Re:Facebook Down? by londons_explore on Thursday April 08, 2021 @06:15PM (#26744495)
I guarantee you that there are 100+ people who could take Facebook down for 24+ hours if they went rogue.

For example the people responsible for the bootup scripts of Facebook infra could sneak in a "0 0 1 * * /bin/rm -rf ${TEMPDIR}/*" into crontab... They'd set the commit message as "clear out temp monthly" and it would get deployed across the entire fleet till in the first of next month every disk at Facebook gets erased because TEMPDIR isn't defined...

I guess they have enough pending stock to deter them...

Re:Facebook Down? by viraptor on Thursday April 08, 2021 @09:43PM (#26745891)
This wouldn't "delete" Facebook or many much smaller companies. It would result in maybe a small outage and get restored immediately in most cases. It's also an infra change you'd need across many systems - this isn't possible as a single change "across entire fleet".

This is not how non-trivial services work.

Re:Facebook Down? by austinheap on Thursday April 08, 2021 @06:40PM (#26744707)
That’s...now how any of this works. You can’t just change integrity-bearing things without FIM systems kicking in. And you’d need collusion to get something mainlined that would bypass that.
Re:Facebook Down? by londons_explore on Thursday April 08, 2021 @06:42PM (#26744720)
You don't need collusion - just a code reviewer not paying proper attention.
Re:Facebook Down? by tracyhenry on Thursday April 08, 2021 @06:35PM (#26744656)
How though? Every past author of that script would be notified of such a change. It'd be insane if all of them would pretend they didn't see it and accept that change.
Re:Facebook Down? by londons_explore on Thursday April 08, 2021 @06:41PM (#26744709)
It would be hidden amongst a big refactor, and it would have a bunch of unit and integration tests that all work correctly (because they do set the environment variables correctly).

Writing code that appears to do one thing and actually does another is very doable - you only need to hide one malicious line amongst thousands in a code review, while the reviewer needs to inspect every line.

Besides, the vast majority of code reviews are 10 minutes or less.

Re:Facebook Down? by tracyhenry on Thursday April 08, 2021 @07:28PM (#26745034)
I think you oversimplified the code merge process at Facebook. Their internal code review system (Phabricator) would automatically add tens of reviewers to your PR because you changed an important script authored by those people. Also, there are linters that would call out the use of potentially hazardous commands. Actually deploying the thing is another process that requires jumping through a series of other review hoops.

p.s. I had a short internship there.

Re:Facebook Down? by ta20210405 on Thursday April 08, 2021 @06:41PM (#26744713)
Facebook servers do not have cron installed.
Re:Facebook Down? by londons_explore on Thursday April 08, 2021 @06:44PM (#26744734)
Example using widely understood tooling... But there are plenty of other one line changes with similar impact