Our company is actually building Git for Data and Github for Data. We have an open source database called Dolt which combines the commit graph of git with the relational tables and SQL of MySQL:

https://github.com/liquidata-inc/dolt/

Then we have a DoltHub, which is Github for data:

https://www.dolthub.com/

Dolt lets you version, branch, and merge your dataset so that you can collaborate on it with others. Dolthub lets you share your dataset with the world, submit PRs, fork other people's repos, and lots of other analogous features to Github.

> A mental retard who is clueless not only about current events, but also has the IQ level of a rock. "Dolt" may be the most sophisticated insult in the English language. Dolts commonly populate such stereotypes as jocks, nerds, fruits, bookworms, and dorks.

https://en.wikipedia.org/wiki/Git#Naming

This is a particularly good question since it's recently been shown that even neural nets trained on real data often pick up substantial, predictable dataset biases.

Practically every single-dataset-trained CNN seems to pick up stylistic quirks in the photos or labels it's trained on. The most visible result is that the CNNs perform better on same-dataset test examples than they do in the wild, sometimes vastly better. More startlingly, it's possible to work backwards from this: the training source of a "finished" CNN can be discerned by looking for certain types of error, and adversarial examples can be predictably constructed based on training source.

Tagged imagesets undoubtedly have stronger and harder-to-remove 'fingerprints' than text data like addresses, but I'd be shocked if the problem was nonexistent for text. My first reaction to "synthetic sensitive user data" for ML is to worry about winding up with systematic errors coming from the generation scheme.

It's a technique we've used in the DoD a long time and it works ok when everything is perfect. There are a lot of boundary problems like being able to troubleshoot bad data if you are doing your initial analysis with the transformed data, having DS's actually grok the problem-set since it's abstracted etc...

Edit: It's worth noting that this is a technical solution to a policy/legal roadblock. As organizations mature into better data governance, they are pushing more fundamental changes to governance that gets at these problems where solutions like this will no longer be necessary. For example, hiring data scientists into the groups that have access to the raw data (in our case hiring data scientists and giving them security clearances).

The use cases that this addresses are ones where you cannot train models in the same system/space/network as where the data exists.

G: Q -> (X,Y)

where Q is some prior from which you are sampling. If they are not invertible then you straight up cannot get P(X,Y) out of the generator. Even if it is invertible getting P(X) requires integrating out the Y which might be infeasible (since the model is not integrable and is sufficiently fast changing that you need very, very many samples).

Say you use regular expressions to extract sensitive data from standardized, but numerously varied, form documents. The pieces of information extracted are very common classes of data: first name, last name, dates, physical locations.

During the extraction process you can save the complement of the extraction (the "leftovers") and insert generated data at the extraction points. Also, because you've extracted the actual sensitive data, you can exclude that from the set of values used for generation, if it's practical.

Sometimes people get caught up in the math and theory that they fail to see the practical solutions.

Which is part of the reason for the high failure rate.

Good governance and data access for health data is a very hard problem. Good labeling is also hard/expensive in this space.

So there is an incentive for people wanting to do ML/AI without solving above to try any kind of shortcut they can think of. This incentive doesn't help solve any real problems.

The classic solution to "too little data" is use a simpler and/or less discriminating model. It's still the only one with a good track record.

Kind of, transfer learning is an important tool that people should use more.

People already use it quite a lot. More importantly they misuse it a lot. I'd be less concerned with increasing the usage, and more concerned that the people using it understand the implications and trade offs.

Just because it may have some drop in utility today from real data, there are all sorts of scenarios where that’s outweighed by speed and ease of working with fake data rather that tied up in red tape real data.

See for example: https://hazy.com/blog/2019/12/09/data-science-on-test-data

Also, synthetic data can be augmented, rebalanced etc. Which is why in future it may well perform better than real data for data science work.

For example, think about this person does not exist and then apply that to business data.

Disclosure: Hazy cofounder. We’ve been doing smart synthetic data for a few years now.

There are a few areas of current practice that have this feature: a) the arguments & evidence for it being worse are pretty simple and b) the arguments & evidence for potential benefits are either weak or very convoluted. This is never a good sign.

I think this happens mostly because the reasons these things are being done are for the most part not technical, but the technically oriented people involved don't like to think about that way, and would rather talk about technical solutions - but that is operating at the wrong data.

The business & cost cases behind not doing this "right" in some abstract sense are pretty clear too, though. I wish more people would just be clear about this, and spend less effort obfuscating and more in clearly quantifying the cost of these workarounds.

Any time you hear someone starting off by saying things like "we don't really need good labels", "this synthetic data will be better, actually", "we'll use transfer from X because it's already done most of the work", etc., well what follows is quite likely to be good fertilizer.

Note, I'm not saying these approaches don't have value, just that there is an awful lot of magical thinking going on around it, and a lot of failures due to that.

I totally agree that the business need is the driver and that people miss the imperatives if they look at it purely from a technical or mathematical lens.

In a sense, synthetic data is the least bad actually viable solution. The democracy of privacy / data agility :p

My issue is not that people just miss the imperatives, but that they also misapply effort because of it. Accept there is a cost and try and quantify the impact. Make intelligent risk management decisions based on that. Sometimes that decision is "this is unlikely to work, what else can we do".

Also this for a practical use of synthetic data.

Of course, that would still be hairy as you'd have to ensure your ML still performed on real data sets. All the clean data would do is allow you to write unit tests of sorts for your ML with more confidence than the all-too-common unclean real data.

No idea, just making stuff up. Interesting thought.

That's what I'm guessing. It could be used for AI stuff but also other useful datasets that are closed off or require special access.

I'm guessing a lot of defence contractors and other heavily regulated industries (ie, healthcare, insurance, pharma, etc) have similar problems in the dev process of not having access to real data. This was the leading pitch:

>> Data is valuable for helping developers and engineers to build new features and better innovate.

That's just my guess though.

SDR captures PCAPs Microphone captures Public tax records Road data Flight plans etc...

Just anything and everything on one service with the limitation that it's all open data. No license agreements or legal restrictions.

It would probably never fly, but it would be amazing.