On the other hand, the lack of data for independent researchers may encourage the development of low data techniques which is much more exciting in the long term since humans are able to learn with much less data than required by most machine learning techniques

I think an exciting area that can innovate the lack of data is domain randomization, and synthetic data generation.

Slides from Josh Tobin is a great introduction: http://josh-tobin.com/assets/pdf/randomization_and_the_reali...

http://josh-tobin.com/assets/pdf/BeyondDomainRandomization_T...

And a really cool project implementing synthetic generation of text in images: https://github.com/ankush-me/SynthText

How it that useful for subsequent learning? The output is random words that doesn't even forms phrases or sentences and has no relation with the image.

Arguably humans have a lifetime of data which was used to develop a model of the world that is amazingly efficient at interpreting new data.

Humans can transfer learn across domains because we can draw on an incredible wealth of past experience. We can understand and abstractly reason about the architecture of problem landscapes and map our understanding into new spaces.

That isn't even counting our hardwired animal intelligence.

Humans learn efficiently, but that has nothing to do with having a lifetime of data I believe. Humans have a lifetime of data, but you can easily parallelize the model such that it takes in more data than a human can in his/her lifetime, and still humans are still the state of the art.

Or our entire evolutionary history of data.

...which fits into a size of less than 700Mb compressed. Some of the most exciting stories I've read recently for machine learning are cases where learning is re-used between different problems. Strip off a few layers, do minimal re-training and it learns a new problem, quickly. In the next decade, I can easily see some unanticipated techniques blowing the lid off this field.

I’m not sure our genetics encodes all the physics of being a person. A human brain is so complex we’re not even close to simulating it on silicon

It indeed strikes me as particularly domain-narrow when I hear neuro or ML scientists claim as self-evident that "humans can learn new stuff with just a few examples!.." when the hardware upon which said learning takes place has been exposed to such 'examples' likely trillions of times over billions of years before — encoded as DNA and whatever else runs the 'make' command on us.

The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself.

It's really intuition based on 'natural' evolution, but I think you don't get to train much "intelligence" in 1 generation of being, however complex your being might be (or else humans would be rising exponentially in intelligence every generation by now, and think of what that means to the symmetrical assumption about silicon-based intelligence).

"The usual corollary (that ML should "therefore" be able to learn with a few examples) may only apply, as I see it, if we somehow encode previous "learning" about the problem in very the structure (architecture, hardware, design) of the model itself."

Yes, and they do. They aren't choosing completely arbitrary algorithms when they attempt to solve a ML problem, they are typically using approaches that have already been proven to work well on related problems, or at least are variants of proven approaches.

Indeed, but what's the magnitude?

The question is, how much information is encoded in those algos (to me, low-order logical truths about a few elementary variables, low degree of freedom for the system overall), compared to how much information is encoded in the "algos of the human brain" (and actually the whole body, if we admit that intelligence has little motivation to emerge if there's no signal to process and no action to ever be taken).

I was merely pointing out this outstanding asymmetry, as I see it, and the unfairness of judging our AI progress (or setting goals for it) relatively to anything even remotely close to evolved species, in terms of end-result behavior, emergent high-level observations.

Think of it this way: a tiny neural net (equivalent to the brain of what, not even an insect?) "generationally evolved" enough by us to be able to recognize cats and license numbers and process human speech and suggest songs and whatnot is really not too shabby. I'd call it monumental successs to be able to focus a NN so well on a vertical skill. But that's also low-order low-freedom, in the grander scheme of things, and "focus" (verticality) is just one aspect of intelligence (e.g. the raging battle is for "context" as we speak, horizontality and sequentiality of knowledge; and you can see how the concept of "awareness", even just mechanical, lies behind that). So, many more steps to go. So vastly much more to encode in our models before they're able to take a lesson in one standing and a few examples.

It really took big-big-big data for evolution to do it, anyway, and we're speeding that up thanks to focus in design, and electronics to hasten information processing, but not fundamentally changing the law of neural evolution, it seems.

If you ask me, the next step is to encode structural information in the neuron itself, as a machine or even network thereof, because that's how biology does it (the "dumb" logic gate transistor model is definitely wrong on all accounts, too simplistic). Seems like the next obvious move, architecturally.

humans dont start with a random scrambled brain

Is that in a csv.gz I can torrent somewhere?

Are you referring to empiricism?

Transfer learning for the win.

Low data techniques are just another name for algorithms/equations. Dijstras algorithm required 0 training graphs to make.

Any other kind of method will get killed by low statistical information in the data (can't get blood from a stone)

Agree with your first statement and disagree with your second; I don’t think the former implies the latter.

I think there’s a lot of room to be clever with encoding domain-specific inductive biases into models/algorithms, such that they can perform fast+robust inference. Exploiting this trade off as a design parameter to be tuned, rather than sitting at one of the two extremes is potentially going to generate a lot of value. And this is highly under-appreciated currently since most people are obsessed with “data”. I’m willing to bet that this will become big in a few years when the current AI hype machine falters, and will serve as a huge competitive advantage.

These types of techniques are already big in certain fields. E.g., in fluid dynamics and heat transfer, "dimensional analysis" is frequently used to simplify and generalize models. Sometimes models can be nearly fully specified up to a constant of proportionality based solely on dimensional considerations. Beyond what is typically seen as "data" the information here is a list of variables involved in the problem and the dimensions of the variables.

As far as I can tell "dimensions" in this sense are a purely human construct. For two variables to have different dimensions, it means that they can not be meaningfully added, e.g., apples and oranges.

This would be a great area, IMHO, for the government to step in and fund an initiative to provide huge, rich datasets for anyone to use for ML research.

wrt the data point, to be fair most research is still coming out of universities where students have access to the same data as anyone else. So from a research perspective it's not a huge deal, much as with compute industry can scale up known techniques while individual researchers do more interesting stuff.

So if I understand correctly, to reformulate in my own words/views:

while the "big data" (datasets) formed and thus owned by big-tech, big-ads, big-brother, etc. may be instrumental to build at-scale solutions for real-world usage (for profit, knowledge, control, whatever actionable goal),

fundamental research itself, as done in universities, can move forward without these datasets: using what's publicly available is enough.

Did I read this right? It would effectively add much needed nuance to the common perception that big data is necessary to train innovative models, that there might be some sort of monopoly on oil (data, the 'fuel' of ML) by a few champions of data collection.

It's not exactly true that research institutions don't have access to the same big datasets as companies. For example, I took a course that involved tracking soccer players using videos provided by a streaming company that specializes in amateur soccer. They promised to give us access to their internal API under an NDA, which they wouldn't have done for just anyone.

On the other hand, they never actually gave our API keys the necessary privileges, so in the end I just reverse-engineered the URL scheme of their streams and scraped them. Many datasets used in academia are just collections of publicly available data (e.g. Wikipedia, images found by googling), optionally annotated for cheap using Amazon Mechanical Turk. Experimenting with that kind of data is also open to independent researchers. You don't need to work at a data-hoarding company if you can get what you need by scraping their website.

yep, you read that right. Source: I am a PhD student at Stanford at the Stanford Vision and Learning lab (http://svl.stanford.edu/) and read a ton of AI papers. The vast majority of papers are done with datasets anyone can just download / request, as far as I've seen.

personally, without affiliation to a university, I have a hard time downloading the datasets through my slow home connection. I live in a city with a university, I explained the situation but they won't let me download a dataset even if I pay, ... only when I enroll. Instead of just selling the shovel, they want to sell me the wheelbarrow too.

I succeeded one time in convincing the guy behind a desk in an internet cafe, so I could bring my HDD and download a dataset in a calmer time of day, and throttled so it wouldn't disturb other customers. This went without any problems for the other customers in the internet cafe. When I asked again a few months later for a new dataset, they no longer wanted me to do so...

There seems to be no download by mail service (and I only get people forwarding me to google cloud products etc, which as a European is so financially out there with automatic balance deductions and non transparent pricing schemes, I would have no qualms using GCP or others if they ran a prepaid alternative for people who refuse to take on risk)

All of which is very satisfying! Thank you for the uplifting view.

A lot of research data sets are publicly available, but many researchers based at universities have relationships with private companies where they can get access to data or other resources useful for research (e.g. Google has a big room of robotic arms generating data for pick and place tasks).

There is still plenty you can do with a reasonable personal budget, however.

> colorizing b&w photos

You will have unlimited training data. But its very difficult task even for humans. Its like trying to reverse a hash. Also a lot of information is lost when you store a color digitally.