Most data is def crap lol.
I don't see few shot, one-shot or no-shot getting anywhere close to standard supervised learning for anything practical. It really doesn't make sense in production settings at all.
You have a function that you want to learn, let's say mapping between an RGB image to a segmentation map. For most applications you're never really in a situation where a production product is dealing with visual scenes/objects it has never seen before. In a factory, in smart stores, in cars, AR scenarios like I just don't see it happening. And then if this case is removed, I'm thinking, ok so when can I get good enough results from a tiny dataset? Machine Learning isn't magic, you're trying to learn a function with 100 million parameters using a dataset, I just don't see the math working out. More data provides better results, it inserts more information to create a more relevant mapping function from input to output.
Third point is great! As long as the models are somehow based on real-world scans I think a lot of good can come from this. The funny thing is, there is so much bias in networks trained today precisely because the data captured is usually small and captured from a specific area/population/setting. If you had a great synthetic data generation engine you could at least generate equal representation of gender, age groups, ethnicities, ... etc.
Overall great points!
1. Totally agree with the point about data science talent being one of the bottlenecks. And in that area, the big tech companies already have the best & brightest workforce. Most data is crap is a bit of a stretch, but agree that a lot of pre-processing is required. But more and more platforms and/or services company are filling this gap.
2. 100%. This is a short term problem.
3. Agree to some extent. You can span latent spaces on top of an initial batch to overcome scarcity and use GANs to scale up. Data augmentation is not perfect but solves a lot of problems in production. Also specifically as it relates to computer vision, it's "easier" to know what the real world looks like and try to replicate it. It's hard in practice but the assumptions are mostly agreed upon.
Anyone in this space is well aware that the benefit of big data isn't just the amount of data, but that's it's a real representative sample of the type of data you are actually going to need to work with. Big data solves the problem of people being bad at creating simulations. To suggest simulations as a solution to big data is kinda getting the relationship backwards.
Why? I think it's strange to believe the opposite: that something as simple as a computer program designed by something as simple as the human mind definitely should be able to adequately simulate the complexity of the real world.
I'm not sure the second is even a logical possibility, never mind a practical one.
Most data is crap. So the mountains of data that are supposedly an advantage, in China or in the vaults of the large corporations, are not fit for purpose without a tremendous amount of data pre-processing, and even then... And that means the real chokepoint is data science talent, not data quantity. In other words, in many cases, the premises of this statement should be questioned.
Secondly, a lot of research is focused on few-shot, one-shot or no-shot learning. That is, the AI industry will make this constraint increasingly obsolete.
Thirdly, with synthetic data, it is only as good as the assumptions you made while creating it. But how did you reach those assumptions? By what argument should they be treated as a source of truth? If you are making those assumptions based on your wide experience with real-world data, well then, we run into the same scarcity, mediated by the mind of the person creating the synthetic data.