Lots of data scientists, ex-statisticians, analysts, etc. are pressured with "you should use git". It's generally a noble thing, and the right one -- these folks often otherwise don't use git enough for their code.

It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed.

I'm not saying what happened here is excusable -- it's not, it's ridiculously bad. But I can see a few ways how it'd happen.

To me the equal or better question is why does this person have PII in the first place? It's not necessary to do analysis. Someone should have masked or removed it before this person ever got their hands on it. There's no way a PII dataset is needed for training.

> these folks often otherwise don't use git enough for their code.

Or at all. This is actually a fairly common problem among certain types of research scientists who think software engineering is "peasant work".

I remember a friend of mine working at a research organization where several researchers lost months of work due to an unscheduled server reboot. Turns out when you log into ephemeral containers and pretend they are VMs, things go poof.

I read a post on Reddit's legal advice subreddit of a data scientist having their apartment robbed, and part of it was saying they'd lost all their code and currently working on projects.

As horrible as the robbery was, I was screaming inside "WHAT ABOUT VERSION CONTROL?!??!"

The worst story I read was of someone working on his PhD thesis for 2 years, and then he forgot his laptop in a bus and everything was gone because in 2 years he didn't store the files anywhere else. I personally met a teacher who stored the only copy of her students' final exam submissions on her everyday thumb drive without backup.

For many people technology really is magic.

That’s a really sad story.

If he doesn’t come from a programming background version can seem foreign.

But like ya know… Dropbox. I’d be more concerned with my hard drive corrupting than anything else

I worked at a company which had a bunch of VMs for their data science teams. They replaced them with containers, then suddenly panicked when they realized that the containers would lose state when the admins of the container hosts applied security patches and rebooted or removed the outdated nodes. (The old VMs were rarely, if ever, patched after the initial boot, which became a compliance problem.) It turned into a monstrous issue where the data scientists wanted 6 months of continuity, the data platform SREs wanted 30 days and the container host admins wanted 15 minutes.

I think they might still be deadlocked to this day.

I love hearing these kinds of stories just to remind myself how not dysfunctional my own company actually is.

I know plenty of companies that rarely apply any patches or do OS upgrades. I'm talking servers with 4 to 5 year uptimes, still running Ubuntu 16.04 or worse. They don't want to reboot because some dude who left 4 years ago set everything up, often in a non-standard way, and nobody is quite sure how it works. They certainly don't want to be blamed when production goes down. I did a contract job for a very large company that was running a 6 year old distro on a "critical" server and was afraid to do anything beyond changing a password. It's easier to have an outside person do it, so they can blame them when it gets screwed up.

Wouldn’t it be possible to automatically backup the data before reboot ?

Docker offers persistent volumes, so my 30 second solution would be that

That assumes a willingness to learn enough about Docker to understand how you configure your container to do what you want. That can, in some organizations, be too much to ask of researchers. That's the core of the problem.

I believe that was asked for by the SREs- e.g. Tensorflow supports checkpointing to disk and restoring progress- but the ML training software used by the data scientists did not have this feature.

Correct. This is a common failure mode... The easiest way to build a training set for a system is to take some live data, salami slice it, and store it in a static location.

... that's also a short path to dropping a user's PII in the clear.

Use git but don't use github.

git alone doesn't protect you against accidentally deleting your code directory and whatnot, and GitHub is the most accessible remote git offering right now.

Isn't this what backups are for?

Your operating system might. But it doesn't protect against your machine dying. That's a different problem with different solutions. You could self host on a different machine...

Is github the most accessible? It is the most popular and best for discovery but a discussion around if gitlabs or bitbucket being more accessible could be had.

I was thinking: what about just some different PC where you got ssh access. Yes this might not be accessible enough. So the next thing could be gitlab. They offer free private repos

> It does however come with this risk, that they let data, or Jupyter notebook output or etc. get committed.

You make that sound like it's generically a bad thing. Having the data you used and tested with is good in many ways. Having it in the same git branch and repo as everything else is suboptimal but usually a lot better than not safekeeping it at all.

I would say not just why are you committing it to git, but why are you using it for "employee training"?

You should not be using real data with PII for training exersizes.

Hopefully that's what they mean by:

> We are additionally putting in place more protocols and access controls to avoid any possible future data exposure and limiting access for employee training use.

They're not, they're 'just' letting employees use real customer PII when training.

They screwed up by allowing that. The employee screwed up by committing it to git and then pushing to a public repo.

The employee wouldn't have been able to do that if they'd enforced using fake customer data for testing/training.

The unfortunate thing about startups is that a lot of them are this fast and loose with PII. Incentives are low to do better, and the tools that make it easy to do better cost money.

This isn't to excuse Adafruit; it's to remind everyone that the hot young startup you just signed up for is probably keeping your signup information in a mysql database that everyone in the company has access to right now with a plaintext password thumb-tacked to the one office wall they have.

Yep. And there’s a good chance the company’s production database is on employee laptops so they can test features with real data sets.

When a stranger has their laptop stolen on a bus, who knows what data was on it. Fingers crossed most people have FileVault turned on these days.

> The unfortunate thing about startups is that a lot of them are this fast and loose with PII

Unless it is a HIPAA-compliant startup. Then scrubbing PII is priority number 1.

I know of business units IN government who explicitly ignore compliance. They sign off claiming they "accept the risk". It's worth your job if you push compliance too hard with them.

I thankfully have never had the displeasure of working with a business unit that explicitly ignored compliance with regulations.

i know they exist. i just havent, personally, had that experience, thankfully.

When you try to tell a business unit they can't use live/prod data then whine to their director who complains to their deputy minister, at which point the hailstorm of shit turns around and starts falling down on those of us who are "blockers". Don't get me started on "the business signs off on accepting the risk".

There's nothing particularly wrong with using git for PII, as long as you use it internally on private machines or a private LAN.

But it's a bad idea to put PII on github which is apparently what happened.

I think that even just on private machines, this would make some types of legal compliance needlessly difficult. If you ever need to delete that data, for example to comply with a corporate retention policy or in response to a request from an individual in a jurisdiction that requires you allow doing so, you would need not just to rewrite history but also to ensure that history is rewritten in every clone that any employee has ever made of that repository; there might not even be a record of which clones exist.

The clone issue is real, but it's no more complicated than it would be without git: Keep track of who has a copy of this data and where. Whatever system you have in place to solve that problem can also be used to manage the cloning process.

As for the individual repos, here's the procedure:

  cd <repo>
  rm -rf .git/  # DON'T TRY THIS AT HOME unless you know what you're doing.
  git add .
  git commit -m "Brand new repo"

Do not do the above unless you completely understand the ramifications. But those ramifications might be precisely the ones you want for legal compliance.

Please never do this -- not least of which because each person who does it will end up with now completely divergent histories and repos from each other.

There are various tools to properly filter history, e.g. `git-filter-repo`, but the short answer is as the grandparent commenter said, things get hairy, you need to rewrite history and coordinate. It's not a situation you should hope to get yourself into...

Might as well not use git then

Using git is no big deal, but storing the repo on Github means sharing the information with a third party (Microsoft) even if the repo is kept "private".

What mechanisms are in place to prevent an employee from doing so? We have so many people in the industry now that it's hard to keep up with this stuff. To be fair, Github has all the ingredients necessary to be a giant, gaping security liability. It's a "secure" platform built around software designed for sharing. That sentence alone has a ton of baggage in it. I'm not even trying to diss Github either.

Education/training is really the only way.