Building a large-scale information system is like building a nuclear power station. There are a million ways to screw it up and only a few recognized right ways. If you ignore the best practices, it will eventually destroy your company and harm your users. Twitter have nuked themselves here. How can they come back from this? It sure looks like an insider risk mitigation system would have been money well spent.
I had a fairly high level of Gmail and Gaia administrator access for a while when I worked there, including the post Snowden era. Resetting the password on an account would indeed trigger an audit event, and I'd be asked what was going on. I could provide any plausible sounding reason and that was sufficient, it wasn't really investigated. And that was the right level of oversight because as far as I know nobody with that kind of access ever abused it by making up a plausible sounding reason.
Stopping bad insiders is really hard. Attempting to do it makes most organisations totally dysfunctional. There is one very famous kind of company that combats bad insiders regularly and with huge quantities of systems - a bank. Investment banks in particular. Whenever you read about 'rogue traders' they inevitably had to do a lot of stuff to disable all the various security systems trying to catch rogue traders.
Institutionally distrusting your own employees can lead to seriously messed up IT systems. It's one of the reasons that bank employees are notoriously unable to access so many ordinary external websites, or services like Slack. It's how you can get "administrators" that can't read the logs of the service they supposedly administer. Encrypted messaging services in particular are poison to an org that's trying to stop employees exfiltrating valuable data. Google can just about do a good job of it because it has an essentially unlimited budget, which it spends on rolling its own tools for absolutely everything and integrating it all into one uber-architecture. From an economics perspective this makes no sense - comparative advantage etc - and thus basically no other company can do it that way. They have to buy or deploy open source tools that use a wide array of threat models and security systems but 95% of them will assume a trusted admin. Then try and hack things on top to restrict what rogue admins can do. It's deeply unpleasant.
I'm quite concerned about what that means and what this means, and I'm watching this intently. Probably for nothing; I know this is in the realm of risk we're unprepared for, and can't prepare for. Darned if I don't worry anyway.
Yes, that might be a bad trade in the long run, but history has shown us times and times again that people are bad at evaluating those risks.
We all know access controls and multiple operators are good, yeah. But at the heart of it is still a bunch of linux machines that have to be managed and deployed to. Which as far as I know has no mechanism for check with operator x before running command from operator 0.
- at-rest encryption of the datastores with the content encryption key protected by a HSM. A KMS (key management system) would be the interface to retrieve the key, with access control enabled. An even better solution would be to have the HSM cipher/decipher the data directly, thus the encryption key would never leave the HSM (or the encryption key is also ciphered by the HSM). But performance-wise it is not realistic.
- in-transit encryption from the client to the datastore. No end-to-end encryption more likely thus allowing admins who have access to encryption termination hosts (reverse proxy, twitter backend app, datastore,etc) to read (and maybe alter) the data by doing memory dumps
- access control for datastore operations: allowing only the twitter backend and some privileged users to read/write in the datastores, etc.
Doing end-to-end encryption from the client to the datastore with a key per client is possible but it would make the solution very complex to operate and not performant.
The tl;dr is that they use hardware security modules (HSMs) with quorum-based access controls. Any administrative actions such as deploying software or changing the list of authorized operators requires a quorum of operators to sign a command for that action using their respective private keys.
While this system was designed specifically around protecting customers' private keys, you could imagine a similar system around large databases.
Not necessary
> or filesystem access
Also no
> or ability to modify the fleet.
Not that either. It feel like the conversation around these things is stuck in the far past. Large-scale organizations can and have driven the number of people with root passwords to zero. "Filesystem access" shouldn't be as easy as you're implying and it also shouldn't be of any use, since everything in the files ought to be separately encrypted with keys that can only be unwrapped by authorized systems.
Even the last thing you said about Linux systems starting processes ... even a minor application of imagination can lead you to think of an init daemon that can enforce the pedigree of every process on the machine.
The software has to get there somehow. The images have to get created somehow. The databases need to stay running somehow. At the end of the day they are machines that need to be managed. Just because you don't have people SSH'ing in and SFTP'ing files around changes nothing about that. And I'm not talking about doing that anyway, or any of the other things you keep telling me I don't understand are bad practice (you're wrong).
Hand waving and mumbling 'old tech, newb' doesn't help in the slightest. I've been writing software with a small side of infrastructure management for 10+ years. Not all of us work at FAANG and magically know how things work on that scale.
Thanks anyway.
OK, what about the people who have physical access?
> even a minor application of imagination can lead you to think of an init daemon that can enforce the pedigree of every process on the machine.
Who watches the init daemon?
What about them? Nothing about physical presence should lead to userdata access, nor the ability to act as users, if the application-layer security is squared away. In any case, physical security is by far the easiest of these topics to handle. Keeping people out of buildings is a human undertaking with 1000s of years of solid doctrine.
> Who watches the init daemon?
Another important question! If you don't know what's running on your box, you really don't have a security story at all.
https://cloud.google.com/blog/products/gcp/titan-in-depth-se...
Presumably this database runs on some machine? And this machine was logged into in order to install and setup the database?
Encrypted rows of data are meaningless to an "admin" that can query to its heart's content but will never be able to decrypt the result set. On the other hand, the layers on top (such as the web-tier that emits the plaintext) may have the keys to decrypt, but lack the privs to run around in the database; from that level, they must pass along the user's credentials to obtain user specific content.
Since people don't search by content on Twitter (afaik) and only 'meta-data' indexes are used (such as hash-tags, follower, following, date) this is entirely doable for something like Twitter.
There is also 'Homomorphic Encryption', but I'm not sure the tech there has reached acceptable performance levels.
In my understanding once you remove all the layers of abstraction as some point it's a bunch of databases and data stores. Someone has to manage them. Why wouldn't a breach of those users be able to do whatever they want?
And a higher level, someone is writing the code to implement such a stringent access system. Why wouldn't a breach of those users (or a rogue employee) be able to accomplish bad things?