Traditionally, 1) we start with big, established companies in tech. 2) A few people, dissatisfied with the trammels that big companies place on their employees, leave, found a new company, and then, 3) through superior grit, gumption, execution, flexibility, and speed, steal market share from the big companies. As the new company grows, 4) it accumulates organizational scar tissue and is slowly infiltrated by incompetent careerists, and finally, 5) becomes the very company that the founders set out to create.
We've been going through this cycle since the late 1950s, when the "traitorous eight" left Shockley Semiconductor Laboratory to found "Fairchild Semiconductor".
Machine learning might stop this cycle by breaking step #3. Big companies have access to huge data sets that no group of human developers, no matter how talented, can match. Even slow, hidebound, and barely-competent use of these data sets can squash upstart smaller companies.
It'll be like a planet, full of water and life, with a core that cools to the point where it can no longer sustain plate tectonics. Without constant replenishment, the crust seizes up, the oceans boil off, and the life dies. The tech industry will become Mars.
When the economic cake is growing slowly, every last drop of productivity enhancement counts, and ML is doing a good job at shaving a few basis points off basic costs, such as energy (see DeepMind's work on Google's datacenters), transportation (autonomous cars), and human capital (making some simple jobs redundant). That's why everyone is piling into IoT/ML/AI right now.
What ML is NOT doing is creating a new paradigm/market/computing platform. It's optimizing old markets and technologies. It's not baking new cakes.
We'll see a new platform soon. The cycle of human technological progress has been chugging along steadily for a long time and has sped up exponentially in the past 200 years. I'm sure some new tech will come along soon that will allow us to bake more cakes instead of spreading icing ever-thinner on the ones we already have.
> What ML is NOT doing is creating a new paradigm/market/computing platform. It's optimizing old markets and technologies. It's not baking new cakes.
This is simply not true: there are many products/markets (such as automatic translation, speech recognition) that are simply not possible without this progress in ML.
In translate it replaces a naive translator's job. In speech recognition it gives you a new-ish interface to systems.
In prediction and personalization it gives you what you likely want. (Local weather, local traffic info, local news, in the language it knows you understand, in the format you prefer, at the time it think you most likely want it, etc..).
It puts some "smart" into things. But translate is just a smart dictionary, it's not a real translator, for that we need stronger AI (something like the Jeopardy playing Watson + Google Knowledge Graph / FreeBase + language translation + it should ask questions if it doesn't understand something).
ML is amazing, but it's just a slow march toward more and more adaptive smarts (general intelligence) in a box (hence artificial). And there's probably a tipping point for that. When it can start to learn, or program itself, blablabla... ( https://intelligence.org/2013/04/29/intelligence-explosion-m... )
What are computers good for? They are just replacing a calculator, which was just replacing some manual calculation machines, which were just replacing mental calculus.
A "naive translator's job" cannot translate arbitrary sentences for free and instantaneously. We are not far from real-time (and quality) translation that will make it possible to talk to somebody in a foreign language and have everything translated on the go (there's already a feature that replaces text in a foreign language in the image you're viewing).
I don't know :) I think I'm quite satisfied with the level of disruption ML is bringing to the world. And I feel people are constantly pushing away against this ("This is not real AI!") every time we start to understand how these things work.
Reinforcement Learning is the case of a program teaching itself (without training data or instructions) but I still, some will argue it is not AI because "it's just maths and engineering hacks", I guess.
I'm not saying it's not real AI, I'm saying I can't wait for the time when these separate components/models can be "synergized", when one big system can be trained for multiple tasks, and when it can train itself tasks.
FYI there is a company in Australia that already does this, well, you get discounts if your device shows that you're a safe driver or something...
Especially if you're looking at error rates; 99% accuracy is very different to 95% accuracy is very different to 90% etc.
When seen in this light, it would be surprising if ML were a disruptive technology, since it falls more into the "sustaining" category of innovation.
Most of the large tech companies were built on technologies that no-one expected to become world-defining companies at the time.
With all this focus on ML/AI, it would be very surprising if a small ML/AI shop became huge, rather then getting bought or replicated.
Almost by definition, the big companies of the next era will come from the spaces that large companies are ignoring.
My entire point is that they aren't ignoring any spaces, they have all future spaces accounted for by having their tentacles in literally everything possible that could be future tech.
Bldinsides come from different industries, so they need to cover their bases from all directions, so that's what they all have done with their projects, products, investments in...AI, Energy, Biology, Transportation, Infrastructure, Waste Disposal, Agriculture the list goes on.
The time of the non-ubiquitous firm is over.
AFGAM aren't software companies, they are innovation conglomerates. I mean why do you think google changed to Alphabet?
Advances and implementations of ML is driving the creation and iteration of products for these companies, as well as helping them run their companies and evaluate where to go next - follow the data as it were.
So it is a critical point that every one of their CEOs has outlined in clear detail over the past few years.
valid question here would be - would a fresh human mind perform that well too? Or may be such performance is conditioned upon having at least several years of experience of living on planet Earth and thus having the models of animals and everything else around been built using billions of images that human eyes would have generated during those years?
> It's well known that existing machine learning algorithms are extremely inefficient learners, which is why you need enormous datasets and computing resources to be competitive currently.
when put into perspective, i.e. compared with the cardinality of the above mentioned set of images human eyes generate and computational performance of the brain - 40K TFlops - computers don't look that inefficient learners to me.
I think it's questions like this that will point the way forward. Humans definitely don't come working at 100% as soon as we're born. Lots of interaction with the environment is necessary to become a functioning person. But at the same time, we still seem to have a real ability to generalize hard-wired in. A human in a small society might only speak with a few dozen individuals growing up, but they don't get confused when they meet a new person and hear a new voice.
It seems to me that lots of data will be necessary to get things off the ground --- but I think the sort of "big data" needed for this will be qualitatively different than the big datasets used to train modern state of the art models. My hunch is that it will involve an agent's interactions with its environment (so it will look something like reinforcement learning).
Also, we are born with very good models (for faces, cognition of human languages + speaking and hearing them, seeing things on Earth, movement coordination, other cognitive fundamentals, like counting, categorization, and so on) plus all of that really well packaged as a starter kit, a do-it-yourself general intelligence, with amazing supervisory framework (parents, peers, society, reflective optimization about one's actions) and reinforcement (emotions, memory coding is waaay too drastically modulated by emotions).
And we're still not sure what else is in there, how all this communicates. What's the operating system of consciousness? (We "know" that consciousness is just a "program", but is it the main thread, is it a scheduler, what's the right mental model for understanding the interaction of brain faculties, components of cognition and consciousness itself?)
It does seems pretty well established that some humans overfit on the particular aspects of verbal communication that we generally call "accent" (not to mention actual lects), even when given access to a much larger and more diverse set of individuals and their utterances to generalise from.
Funny you should mention that...
Deepmind (acquired by Google in 2014) goal: "DeepMind Technologies' goal is to "solve intelligence" Demis has gone one to say elsewhere their goal is to create AGI
Also, the data you need to get there is owned by a few companies. Also those companies hire effectively every great ML PhD on the planet.
It's well known that existing machine learning algorithms are extremely inefficient learners
Right, which is why the heads of research at AFGAM are the pre-eminent minds in the field (Lecun, Hinton etc...) and are actively working on this along with unsupervised learning. These companies are either funding university labs that are developing this stuff or doing it in house.
until of course sheer and blunt volume allows to build more accurate and precise models. To me this is exactly at the core of the current AI wave - the models emerging inside deep representational layers are more complex, accurate and precise way beyond whatever human analytical thinking could have built or even just come up with. My friend just couple days ago was talking about one specific multi-year effort few years ago to constructively build a practical model of sentiment analysis which i in turn contrasted with that recent result from OpenAI of "sentiment neuron" emerging from [unsupervised] learning of next character in Amazon reviews.
Yea that's exactly what I think is happening, and in fact that's one of the core promises of being first into AI - you use it to move faster than anyone else can because it gives you better intelligence and prediction.
Even more so with AGI, which Deepmind (Alphabet company) has explictly stated they are trying to create.
- AFGAM
- Military
The period of being able to build your own competitive AI startup was...what, like 3 years between 2010 and 2013?
At the same time other companies, Apple /Salesforce / Amazon / Oracle and several other smaller ones are well aware of threat posed by Google and are always on the lookout for ML/CV startup that can fill the gap in products relative to Googles offering. E.g. Amazon acquired Orbeus around same time Google launched Cloud Vision.
I'm not talking about just Google. Its AFGAM: Apple, Facebook, Google, Amazon, Microsoft
You can add Baidu in there too probably. That's it though.
always on the lookout for ML/CV startup that can fill the gap in products relative to Googles offering
Right, acquire. Not compete. That's the point. You can't actually compete with these companies, like they did in the past with HP, Dell, Nokia etc... the best you can hope for is to be acquired.
There are always new markets which get ignored. Apple, Facebook, Google, Amazon, Microsoft did not compete with HP, Dell, Nokia they essentially entered a new market. Most ML startups don't offer a new product but rather just enable features and acquisition is often the intended end goal.
We expect that DNNs will a fundamental thread to computing going forward. AFGAM has effectively locked up all of the talent and data in the broadest markets already. Not only that they are building and deploying the frameworks that new products will be built on in those other markets. So they don't need to create the product, they just need YOU to use Tensorflow or CUDA or whatever they come up with that runs on GoogleCloudGPU or Azure Compute etc...
What you're missing is that it's not about making a few million as a startup. It's that they are so far ahead in what is the most fundamental game changing, final human technology that I don't see a future where they are unseated.
There is no AI competition against a company that has Tensor Processing Units among the absolutely incomprehensible data store that Google has amassed.... Especially if you are not some world class mathematician or computer scientist. The best you can hope for is to be acquired if you are in the ML/AI space as a stand-alone entity. In most cases they can probably safely ignore the bulk of AI startups.
Also, those TPU's are available for rent.
Nice that TPU's are available for rent, but to what end? If I run something truly innovative on their hardware (TPU), with their software platform(Tensor)...is that actually innovation, will they have the means to just copy it? How do I compete with that?
As always you want to be careful about dependencies, but that doesn't mean rewriting everything.
I use chrome and little snitch is always telling on it. It calls home so much that I have just given up in order to have a decent experience and gave it full connection rights. I don't know how much of the browser Chrome owns, but I guess it is not insignificant. That is a lot of data that I will never have access to... let alone for my family nor on at Google scale.
Netflix uses AWS and competes with Amazon Prime... I get it. I do think that no one knows the whole story.
Getting back to OP... it has to be damn tiring/frustrating to even contemplate competing in AI/ML space with these big mega corps. I simply would not, that ship has sailed.
By the time they copy you, you're already big. I don't really see the problem as long as the tech companies don't hoard the information and spread the knowledge - which they are doing with initiatives like OpenAI and many more.
The big question is, what happens when they invent AI that builds its own programs and exponentially accelerates? Who needs startups at all then?
They not only have custom silicon (TPU), they have the software too. Do you have extraordinary insight|foresight|genius or have you discovered/developed a major breakthrough in AI/ML? If not, good luck competing with these companies. It is a resource problem at this stage, as consolidation happened and there is no going back.
You can download the whole Wikipedia data dump at under 100GB uncompressed (text only - with media is around a TB). The entire common crawl with 3bn pages is only around 250 TB. While the Wikipedia dataset is too large to fit in ram for most people and the common crawl is too big to fit on a single disk, you can process these in your own local cluster quite easily and relatively cheaply.
Honestly I think the real breakthroughs to be made will be algorithmic and I don't believe those are out of reach for "civilians" outside of the tech giants.
You are stuck with 25TBs of wikipedia for the rest of forever. And "academic datasets".
Because the important data is probably not inferable from web links or otherwise semi-passive Internet structures.
That said, I think the missing component is still a good mind theory.
We need a lot of data currently because algorithms don't generalize like people do.
I'm not convinced you need to be a tech giant to be able to make that breakthrough - I think it's a problem of approach, not a lack of data.
It's just not textual.
Last I checked they all work for...AFGAM (or are teaching)
Can you share your reasoning behind this opinion? Humans have invented a lot of tools and technological platforms over the years, what makes you think ML is the final frontier?
The way I see it, consumers (at least right now) are free not to buy into the hype and consume whatever it is that these companies produce. So if prices are too high or quality too low, a new company can enter in.
The real danger I see is the possibility that essentially all consumers will be forced to pay for this stuff through some regulatory capture, in a similar fashion to the state of the healthcare industry in the US.
Barriers to entry at the "top" of industries are generally higher than lower down.
I'm sympathetic to the sentiment.
Consumers might benefit from the conglomeration more than from a lot of independent groups each reinventing the wheel or spending resources on duplicative infrastructure.
There are trade offs of course. In particular, I really don't want one single ML platform to have all my data and know me that well.
So while Tesla has definitely driven innovation here, there are still MAJOR hurdles to them being globally competitive.
Further it took Billions and someone like Elon Musk to be able to get even this far, so in that context if even Musk and tons of government dollars can't get there then it's safe to say nobody can.
A valid concern, as pointed out, is when the large platform providers begin moving into domain-specific markets, leveraging their scale and closing out start-ups/small business. However, I believe nimbler startups will always be able to carve out a successful niche as long as they are closer to the customers.
The difference is where the majority of the "idle wealth" is created. Sure, there will be great business doing consulting work leveraging cloud providers - you can make a good business simply consulting on such things.
However in the end you're simply enriching a large company. Whenever an "amazon cloud consultant" transitions a closet of servers to AWS - they made a bit of money, but the large bulk of the value of that customer is now in a giant corporation's hands vs. being spread out how it was traditionally (either in house, or via much smaller infrastructure providers that generally competed on service without much lock-in).
The lack of competition in the web space is astounding to me, and will be incredibly damaging to innovation. How quickly people forget why decentralization used to be considered such an important goal. Yet it appears the entire industry is rushing head long into this model.
Startups then return to what they are in any other countries. Just a way to make a living rather than the potential path to riches it is in the bay area.
As a side effect you get smaller elites controlling more of the pie. That has other implications as well.
Nothing to worry about, unless you are trying to create these commoditized items, which is pretty much a doomed undertaking.
I can't, cannot, convince the org to bet the future on the ability and moral grounding of two random guys in a garage. They wouldn't even be able to wrap their heads around it, the difference in scales is too big: "You want to give our 10 PB of data to two guys in a garage? What? Get out. In fact, go get a new job, you're done here."
These deals are being signed by people who worry about capitalized words, like Gross National Product or Personal Health Information or Mars. We're talking orgs with tens of thousands of people who have committed their lives to a thing. We're not trying to use Dutch actions to steer 20-somethings in Orange County to the right sushi restaurant. Which is a sweet trick, don't get me wrong. But it's a bit different when we're trying to protect nations and their economies, and maybe just save the world or the human race.
Do you know what the US Government defines as "a large firm" for contracting purposes? 200 people. That's about right. You and I both know that those same two guys working for AFGAM are still the same two guys. But I need to contract with an entity wherein, if those guys I'm collaborating with go south, I have recourse with the Inc to keep the project on track. They can fire the guys, bring in new guys, subcontract some guys, whatever. But I've got to convince serious people that I can maintain a 10 year trajectory in a white hot, globally competitive market.
Enterprise sales takes lots of demos, trials on subsets of data, support arrangements, relationship building etc.
NASA, DoD? FRA, NHTSA? You're looking at a 10 year lead time.
DARPA programs have a fixed, 2 year span, and they aren't allowed to be renewed. It's deliver something in 2 years or fail.
You and I both know that those same two guys working for AFGAM are still the same two guys.
But it's not. Or at least those two people don't deliver a product, they deliver research outcomes, which then takes double the number of people to turn into reliable software, which then takes double that number of people to make deployable and run at scale, which then takes double that number of people to support. And then you need a sales team.
But there are smaller orgs (let's say Mexico, or Spain, or Greece, or just a town in Germany), they might try whatever they think helps them.
Little companies can adapt these brains to niches that are too small for the big guys.
In the space of all the things you would want to do with ML, this is pretty limited.
[EDIT]: I stumbled upon a transfer learning for NER paper a second ago: https://arxiv.org/abs/1705.06273v1 And the results are not conclusive. You can can improve your results in truly data impoverished regimes, but you get diminishing returns as you add more data, so the only way to catch up is to get more data.
So, you may be able to get reasonable performance with fewer samples, but you will still get better performance with more samples, and the more samples you get, the less benefit you get from transfer learning.
This may not always hold, and I wonder if active learning paired with transfer learning would do better, but at the moment it's not convincing.
As a result, you would success in this area is mostly driven by the data. In other words: this is rolling out exactly how you would expect, no?
That's not to say there aren't opportunities, it's just that if anyone makes significant progress in one of those other areas, it's either on the back of AFGAM or will eventually be absorbed by them through acquisition/targeting.
Would you be opposed to joining one of their AI teams? Clearly you believe they are building great stuff...
This is too extreme. It is the last frontier of some set of problems, but it just introduces a new set of problems. This always happens. Even the implausible singularity still introduces more problems to solve.
It's like even with a world class AI team the majors just blow past everyone with their on demand scale, access to data, hardware, people and distribution.
In the past three weeks the products that were shown at conferences (GTC, Build, I/O) would have been 100 different independent companies, with very highly trained, specialized PhD level researchers and developers only 5 years ago. Now this stuff is just baked in to the top platforms.
This has been happening for a while, and I've been saying it for a while too. No clue where to go from here honestly.
edit: The point I didn't make here but is subtext is that, IMO ML/AI is the last frontier for technology (an IMO humanity) so if a few dominate it, its kind of game over for the existence of smaller players generally.