Gosh - I've worked at shops where we handled multi-terabyte images and we'd regularly stream large chunks of that while debugging tools. I've also worked at places where data was king and 125GB of stuff might be a reasonable dispatch of data to help someone debug.

The volume of data is irrelevant - source code is usually teensy tiny and of far more value to companies than, say, three months of livestream chat logs.

I'm not certain what security hardware you're thinking of - but I'm pretty sure I hate it already since it doesn't effectively guard anything while making everyone's lives difficult. For effective corporate security you need 1) data use policies and 2) access control lists - both of those are generally more effectively implemented at an entirely software level.

Yeah volume is a terrible metric to go by. I work as a data engineer and a lot of the time if I am working between environments or when migrating between data centers will have a copy of the data locally that I can write tests against or move to somewhere I can compare it to a running output. This would be possible to do entirely remotely I guess but not nearly as easy. (note I never do this with anything that contains PII)

It is still fraught with problems, while you (knowingly) wouldn't do it with PII, is not all that reassuring, others could, or compromised system could be used to exfiltrate this data, if the only control is just trust on the users behaving well with their access

That fact in general industry the controls on how PII data is accessed internally is so lightly managed should worry everyone

Trying to protect against leaking developers/employees is like trying to protect against lone gunman terrorists: useless. And, if you try anyway, it is likely to cause more annoyance to everyone involved than actual protection (think TSA).

I disagree. Locking down and logging access to raw data like password hashes or payout information to only those who absolutely need it doesn't cause much annoyance and is very useful.

It protects the company against rogue employees (not even strictly malicious, but also curious employees who want to see more than they should). It limits exposure if an employee's account gets hacked (my pet theory for this Twitch hack). And if something does go wrong, logs help track down the issue/leak.

And at the end of the day, there should be a lightweight way to request access. Many times I've seen people request access that they didn't actually need. And most other times they have access pretty quickly.

Note that it was code that was leaked. Preventing developers from leaking the codebase they are working with is outright impossible. Now combine that with a "monorepo" and even the most junior developer has access to practically the entire company codebase and version control history.

And you can try to prevent them from accessing live/real customer data, but the cost is that they will never be able to debug issues in production. Most companies, even very large ones, are just not able to pay that cost. Not to mention that once you have access to the codebase there are a million ways to leak customer data anyway -- it is a lost battle.

Of course, some stuff you can't avoid, especially code leaking. Luckily code isn't usually that interesting or useful to external parties which is the only reason it isn't leaked more.

For the rest of the stuff, there's a sliding scale. In no universe does your average twitch developer need raw access to password hashes, for example.

What with security as it is on these companies, the code is literally the most sensitive information they can hold, specially in terms of value to the company. With the code out, expect lots more high-profile cracks in the coming months...

"your average twitch developer" needs access to the password hashes or at least the code that checks these hashes the moment they need to debug an issue which involves logging in, and from then its all downwards.

Nope, it was code AND data, including the sensitive type (e.g. user payouts).

Adding to your pet theory I think that WFH has led to a lot of people being casual about their workplace security. For example, leaving a laptop unattended at a Starbucks.

This is just a guess but I wouldn't be surprised if companies have to start taking stricter precautions with their security in a WFH world.

This isn't accurate. There are certainly companies that have extremely in-depth Data Loss Prevention toolsets and teams - everything anyone downloads or moves is logged and alerts fire if things look out of the ordinary. Google clearly had tons of data about how Anthony Levandowski was able to exfiltrate lots of info when he left.

The issue that building these systems accurately so they are NOT a constant annoyance is difficult, expensive, and takes a large team to support well.

There are ways to look for anomalous behavior without creeping too hard (even though it's a business's right to view and monitor all network traffic on their system).

If someone who doesn't have a business need to upload lots of traffic begins uploading large amounts of data, you may ask questions. Maybe you kick off a scripted playbook that then checks for increased logins to other privileged systems, or for large transfers of data from internal sources to the user's desktop.

I dont know dude, I work in an enormous company that you 've heard of, and it's impossible for me to imagine how to extract code out. I can't do it, except if I get remote access and film my screen while scrolling.

Anything else is found quickly. I certainly wouldn't even dream of someone extracting the repo.

Really, you can't simply copy files from a code repo you're working on? You work on a isolated workstation, not connected to any external network, where you are not allowed to bring anything other than plain clothes (TSA-style)? With a sizable army of developers all working this way?

And if it's a remote FB/VNC connection, what is preventing you from just recording the screen? Not really hard...

Most companies I've seen could see all their code extracted with one malformed NFS packet. These are "air gapped" systems holding the type of industrial secrets that we don't want to leak to china. Practically the only real line of defense they have is employee screening, which does not really stop the lone man guy.

If the bulk of it is a git repo, it's probably expected that every engineer will download it regularly.

Case against monorepos?

There are much better cases than this; in this case a monorepo makes it slightly more likely to be caught rather than less. (A monorepo can get to Google size and then you can't check it all out at once and it needs bespoke tooling, which can make it harder to pull this off.)

On the flip side while many smaller repos _can_ have independent ACLs, you are very unlikely to set those up until you reach a certain scale -- and then when you reach that scale it gets hard to implement ACLs across everything at once. So your engineers probably all have access to all your repos until you reach a very large size anyway. So the question becomes just "can someone write a for-loop over all of the repo names and check them all out," and it's like, yeah, that's not terribly hard, I as a programmer can do that pretty easily in bash.

Ideal repo size should not in my view be directed at "how do I prevent compromise to the external world," because VCS is not designed to give you the superpower of being resilient around being compromised. Rather VCS is trying to give you the superpower of time travel. So you should probably scope your repo to "what is the unit that makes sense to time travel with?" -- in other words if you are adamant that you have these independent services which operate decoupled and running this one backwards by a year should not affect that one, then those services should be in separate repos. If on the other hand they have some moderate coupling and rewinding this service by 1 year would break the APIs that that service uses to communicate... then those should ideally be in the same repo so that you can coordinate changes between them to their shared protocol.

> So your engineers probably all have access to all your repos until you reach a very large size anyway.

Happens at my company. We have rudimentary ACL but not sure how its implemented because you can find things via explicit searching, or via "organic finding" via links from repo->repo but it won't be surfaced if you just search for code.

You can still have a monorepo and restrict who has access to certain parts of it. You just have to build the tools to do it.

Google, for example, has a small number of subdirectories in the tree that only certain engineers can view (the really sensitive stuff, like the actual ranking algorithms for search and ads) but the build system is setup to allow you to still link against it.

Not particularly - unless different teams are highly focused on certain subsections of the repository. If everyone might have to look anywhere than you'll need to download all the repos - whether that's one or five hundred.

How often do devs delete and re-clone?

Clean OS install or new hardware should both be daily events at even mid sized companies. Because even if it’s once every 2-4 years per developer that still becomes extremely common in aggregate.

I think the tech giants have warped some people's expectations of what a "mid-sized" company is. I work for a mid-sized company where we roll our own ERP system and we probably average about two clean OS installs per year across the entire development team.

Yes, but GP said "every engineer, regularly" which seems odd.

I suspect lots of junior devs will clone fresh, push changes, nuke repo and repeat. I did when i was young instead of syncing state and rebasing.

> Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.

I am trying to recall, but I am pretty sure when I worked in Microsoft Office that a build would pull down many tens of gigabytes of data.

125GB in one day from the build system wouldn't be uncommon!

That's ingress though. Companies should be monitoring and worrying about egress.

Edit: This won't help against a thumbdrive, but that type of thing should be also tracked.

I'm working on a project and just had to repull my workspace after some local corruption. I pulled 1.2TB out of the office and never got an email. I think it's pretty common for places not to monitor egress that closely.

There was a fad for tools that accomplished this in enterprise networks, with much clearer rules for who needs to access what (it was called "data loss prevention", or DLP) and those tools for the most part don't work. This is a harder problem than it looks like.

DLP products tend to be more about scanning the contents of data for sensitive patterns, at least in my observation of the market. There are other products (typically built into SIEM) that do correlation on login events, network traffic and whatnot to detect anomalous behavior.

I’ve worked on a lot of DLP projects in big enterprise, and I have a very dim view of the entire category of product. A lot of their functionality is just magic black boxes, that unsurprisingly achieve very little. The primary motive for deploying them is not that they’re particularly effective, it’s so that you can tell auditors and other scrutineers that you’ve got a “DLP solution”. The idea that you can grant people access to huge quantities information, but then very strictly control what they do with it is fundamentally flawed. Especially on networks that require large amounts of in and outflow for BAU. Even the most tightly controlled data in the world cannot be protected from an inside leaker (or adversary who has taken control of an insiders access), because it runs into the same “analog hole” issue that DRM products have.

My company has this. It encrypts any file touched on USB. And other software logs every app run. Prevents casual copying but easily circumvented. But somewhere logs may have enough info to trace the source of leak I guess.

These tools (DLP) have gotten better with app migration to K8s, since traffic can be watched prior to encryption in a standardized way. Just an FYI….

The enterprise DLP tools were deployed fleetwide as agents and at network choke points; getting access to the raw data wasn't the problem.

Thank you for mentioning this. I always had a gut feeling that it seems like an extremely hard problem to solve in a sensible way.

> It something I would expect security hardware to have automatically stopped. Even an employee shouldn't be able to download 125GB of stuff without flipping a safety switch somewhere.

Remember that Twitch handles streams. Good luck implementing this without having all sorts of false alarms everywhere.

Plus, you don't have to exfiltrate 125GB in one go.

I feel like once you have it pulled downm, it would be as simple as an upload to s3 (which wouldn't trigger any flags), then making the bucket public whenever you want. Hell, S3 used to (still does?) support being part of a torrent swarm...

Why would that help? They just have to accumulate work over a period of time and then 'lose' their laptop.

That's 6.25GB/day over a 20 day working month. More time, less data per work day, harder to detect.

And it might be disguised as a video stream coming out of the video streaming servers.

But it could also be a 128GB thumb drive plugged into the system somewhere.

> And it might be disguised as a video stream coming out of the video streaming servers.

Just log in to FB messenger or Discord and egress it as small data chunks that way. Lots of people have private chats on work computer for practical purposes.

Discord allows for bots, so you could easily write a script to chunk data and egress, and another to re-assemble.

ML engineers / data scientists are regularly moving terabytes of data around at Amazon.

Indeed , how could this happen, really curious.

So let's say someone with access to all GitHub repos gave the password to someone else, maybe then it was downloaded from another machine?

Or someone stole the credentials and downloaded from another machine?

Or someone got access to such a machine?

It's it not possible to prevent these cases?

How long does such a download take?

Cue "Don't check payment receipts into git" discussion - although I strongly suspect this hack wasn't just about acquiring appropriate credentials and then running `git clone`. It sounds to me like a backup service was compromised.