> CASP14 #s just came out and they’re astounding—DeepMind looks to have solved protein structure prediction. Median GDT_TS went from 68.5 (CASP13) to 92.4!!!! Cf. their 2nd best CASP13 struct scored 92.8 (out of 100). Median RMSD is 2.1Å. I think it's over https://predictioncenter.org/casp14/zscores_final.cgi
[0]: https://twitter.com/MoAlQuraishi/status/1333383634649313280
`.cgi`... we've come full circle
Much like other fields, I do begin to question the academic structure to making advances. It appears something is rotten in the state of academia. Oddly it's academia doing incremental improvements to existing methods but industry making novel leaps and bounds... The other major case in point being NLP
A better comparison would be to national labs, but they are tasked with projects that make no sense for industry to tackle.
The system is working as intended, all players are needed. The team at Alphafold busted their chops in academia and went on to working on problems they could spend decades on.
We've also seen a rise of VC-backed research startups (like DeepMind but many others) whose value proposition (to the VC) only makes sense if the goal is to demonstrate a research capacity to get them bought out by a big company, or as a moonshot to out-compete them on an actual product made possible by the research. Investing in these little research startups themselves is also giving companies a way to push research without having to deal with having the researchers as direct employees, and I'm sure makes some of the startup employees feel a bit safer since there's a separation of money and operation influence. One similarity with modern academia is it selects for those who can do good work but who are also good bureaucrats (write grant proposals well, advising politicians, etc), startups have a selection for good work + good at courting VCs. But the startup just needs a few of them, then they can hire people who just want to do good work.
Another thing that makes corporate even better is they can occasionally spin off research developments into products, they can have some nice advances that only come when you try to productize, and among other reasons by not having to bother with external publishing (which takes time + fights with lawyers and business people) they can routinely be 10+ years ahead of whatever the state of the art in academia is.
As for 'needing' PhDs, I'm not sure. Having some can be convenient, yes, but in many cases not necessary. In some fields the only way to get caught up (i.e. no corporation will train you directly) may be with academic foundations but is a PhD necessary or just some relevant graduate work?
As an example N=1 to show a PhD is not needed always, Jeff Jonas formerly held the title of IBM Chief Scientist where he did some state of the art work in entity resolution. He didn't even finish high school.
Further, since your original position was strongly that academia is useless, it is your onus to back up the implication that PhDs are not (generally) needed/useful, and using "well hmm, not sure if absolutely necessary" logic is fallacious and clouds the issue.
When too many of those really stupid investments go belly up at once, it's called a bubble popping, and it is catastrophic to the economy.
> But in part due to the canonicalization of CASP, protein structure prediction effectively has a two-year clock cycle, where separate research groups guard their discoveries until after CASP results are announced.
and further noting
> As I discussed earlier, it is clear that between the Xu and Zhang groups enough was known to develop a system that would have perhaps rivaled AlphaFold.
Finally, and rather crushingly for your thesis, is the points made about the real industrial groups:
> What is worse than academic groups getting scooped by DeepMind? The fact that the collective powers of Novartis, Pfizer, etc, with their hundreds of thousands (~million?) of employees, let an industrial lab that is a complete outsider to the field, with virtually no prior molecular sciences experience, come in and thoroughly beat them on a problem that is, quite frankly, of far greater importance to pharmaceuticals than it is to Alphabet. It is an indictment of the laughable “basic research” groups of these companies, which pay lip service to fundamental science but focus myopically on target-driven research that they managed to so badly embarrass themselves in this episode.
Modern academia exists to further modern academia, no more and no less. I became disillusioned of any search for truth and progress during my time there, it was really just about showing up the Baker lab at the next CASP and getting the next round of grants secured.
If the goal is to just produce researchers that can work at corporate research labs, then I feel we could get more bang for the buck.
If the goal is to do move research in the public good, something needs to change. Maybe it’s the fact there’s too little money out there, and it causes everyone to chase meager grant money. Or that there’s too much competition between groups. Or a million other reasons.
But I’d love to see it fixed and have more faith in public investment in basic science.
https://www.jamesgmartin.center/2020/01/the-intellectual-and...
But lots of other things have also changed that may or may not be causes:
- many fewer tenure track positions
- many more jobs in industry that require or value a 4 year or advanced degree
- obtaining a college degree as a right of passage seen as increasingly essential to you g adulthood
- the rise of data science: more jobs in industry that have access to lots of data and demand scientific rigor
- the rise of private cloud supercomputing (is deep mind) vs, say, a public university's cluster
- the obsession in some foreign countries of getting advanced degrees from American universities, creating essentially a guest visa workforce that is easily abused
- rise of Big Tech which has money to throw at things like protein folding
That said, this is primarily a computational problem, so the advances here might not be applicable to basic research.
Speaking of which, Google Translate was published in 2006, but when did the "learning from data" approach became an accepted idea in machine translation? I think the earlier attempts at machine translation were more about trying to codify grammar rules in software, than doing statistical learning from large text corpuses? I remember in 2002, the approach of leaning protein substructures from data was already the best performing approach in the protein folding problem.
When we discuss Google's input in NLP, the most important contribution is certainly the "Attention is all you Need" paper, which paved the way for BERT and GTP (Alphafold also uses Attention networks, btw)
Which is the same thing as hand-engineered feature stacks being left behind in vision problems, really. The story in every field is more or less "you're not clever enough to engineer good features"; "you might be clever enough to define good symmetries for the feature space in which the features live... maybe" (convolutional neural networks in image problems); "... but maybe not even that" (attention mechanisms).
People were doing a mix of learning from data and hand engineered solutions before this, but this was the first system learned end-to-end, afaik.
There’s a reason these PhDs at Google mind aren’t in academia after all doing the same work
It happens that the team at deepmind is apparently pretty damn good at deep learning problems, so they're going faster than matching academia labs.
It's not to say that academia has none of the problems you mentioned, but it's imo unreasonable to expect that, in a world where both public and private labs exist, only public ones would make advances.
Industry cares about positive results. If you're not allowed to fail, you will be afraid to explore. That's what Academia is. Then, the industry reaps the fruit of that exploration, which is as it should be.
You need to win grants to survive.
How could one predict success or failure in risky project proposals? In both cases you learn something.
PhD programs don't take people who just have a good GPA; you have to have a research record before you're even in the consideration. I've been on an admissions committee, so this is not conjecture.
Perhaps a good analogy are the inventions of the microscope and telescope. They were advances in technology, which then led to advances in science. I don't know if this will have the same effect as the microscope and telescope, but it would be great if it did. It certainly seems extremely promising.
Where's the difference?
(this is a genuine question, I'm not trying to trick you)
You have to realize that corporate research labs had a high level of recognition back in the 20th century. Labs like the Bell Labs, the RCA Laboratories, or the IBM Research, privately-funded, had a reputation that met or exceeded the standard of not-for-profit or public-funded academic research institutions. They made some of the most important discoveries in the electronics industry of 20th century, like the point-contact transistor, the MOSFET, VLSI, or the UNIX operating system. They were considered a part of the academia, many scientists were their employees. It's only the 1980s after their death that people had the impression that "important research must come from academia, industry is for incremental changes." So, I'd argue that the division between industry and academia is large, but actually smaller than people's perception. If you consider privately-funded researches by the industry as a part of the academia, the current situation is totally normal, nothing unusual.
Interestingly, for those labs to exist, being a monopolistic megacorp is a requirement. It appears to me that today's FAANG monopoly allowed the creation of Google Deepmind and OpenAI, perhaps it's simply a beginning of the repetition of history.
The article The death of corporate research labs had an interesting review. I highly recommend to read the article:
* The death of corporate research labs
> https://blog.dshr.org/2020/05/the-death-of-corporate-researc...
(HN comment: https://news.ycombinator.com/item?id=232466722)
To summarize, those great labs existed and made great contributions because of (1) corporate monopoly on the industry, and (2) the pressure from anti-trust laws. First, due to monopoly, the gigantic size allowed the labs to be the center of gravity and to concentrate all talents and projects into a single place, with a huge research budget for basic research. Second, the pressure from anti-trust laws also forced corporations to invent more into basic research to grow the business, because mergers and acquisitions were restricted. In some cases, the pressure from anti-trust laws also made the corporate labs to share their discoveries in a more open manner, examples included advances in semiconductor [1], or the Unix source code.
Note: but as HN comments pointed out, somewhat ironically, the success of corporate labs relies on anti-trust pressures, but not the actual monopoly-busting enforcement. The breakup of Bell caused the death of the Bell Labs.
Finally their decline,
> The more relaxed antitrust environment in the 1980s, however, changed this status quo. Growth through acquisitions became a more viable alternative to internal research, and hence the need to invest in internal research was reduced.
And it turns out that managing a corporate research labs without losing money is a tricky problem to solve. If the researches are too goal-oriented, short-termism will dominate, basic research in the labs will be ignored. Thus, basic research in the lab must be independent. However, a lab too isolated from the business can also cause great loss.
> Research in corporations is difficult to manage profitably. Research projects have long horizons and few intermediate milestones that are meaningful to non-experts. As a result, research inside companies can only survive if insulated from the short-term performance requirements of business divisions. However, insulating research from business also has perils. [...] Walking this tightrope has been extremely difficult. Greater product market competition, shorter technology life cycles, and more demanding investors have added to this challenge. Companies have increasingly concluded that they can do better by sourcing knowledge from outside, rather than betting on making game-changing discoveries in-house.
And the author argued the death of corporate labs decreased productivity.
>> An unintended consequence of abandoning anti-trust enforcement was thus a slowing of productivity growth, because the this new division of labor wasn't as effective as the labs:
> a new division of innovative labor, with universities focusing on research, large firms focusing on development and commercialization, and spinoffs, startups, and university technology licensing offices responsible for connecting the two.
> The translation of scientific knowledge generated in universities to productivity enhancing technical progress has proved to be more difficult to accomplish in practice than expected. Spinoffs, startups, and university licensing offices have not fully filled the gap left by the decline of the corporate lab. Corporate research has a number of characteristics that make it very valuable for science-based innovation and growth. Large corporations have access to significant resources, can more easily integrate multiple knowledge streams, and direct their research toward solving specific practical problems, which makes it more likely for them to produce commercial applications. University research has tended to be curiosity-driven rather than mission-focused. It has favored insight rather than solutions to specific problems, and partly as a consequence, university research has required additional integration and transformation to become economically useful.
---
[0] https://www.eetimes.com/podcasts/six-words-that-built-the-ic...
> Honeywell brought a lawsuit against us and said you can’t selectively choose people to divulge your technology to. It’s too important. And if you divulge it to anyone, you’ve got to divulge it to everybody. They filed a lawsuit, and the government came down on their side. And RCA basically had to open up all of its patents to everybody if they opened them up to anybody.
AFAIK OpenAI is still independent, despite its recent closeness with Microsoft. Deepmind existed and was active well before being acquired by Google. All these to examples prove is that today's big, monopolistic corporations tend to acquire research labs, not that they are a requirement for their existence or successful activity.
It is not always A versus B.
These fallacies are too common.
This is what it's like when someone really moves the needle. And academic science cannot get it's head around it.
And yet, none of these scientists will suffer any career consequences. Their irrelevant work will be healthily cited by all the other scientists who are doing and have done irrelevant work. They'll retcon a story in their lit reviews about how their irrelevant work led to this.
The career consequences are saved for those who had their eye on the real ball for the last five years, but didn't get there first. For them, the comfortably irrelevant will have the gall to ask in accusatory tones: "What have you been doing these last 5 years?".
They pushed the needle because they understood the field and the new tools. This is how pushing the needle works.
In the end software and tech companies might just eat up the pharmaceutical industry as well. - It's all just code at some level.
The Deepmind team did this with ;
"We trained this system on publicly available data consisting of ~170,000 protein structures from the protein data bank together with large databases containing protein sequences of unknown structure. It uses approximately 128 TPUv3 cores (roughly equivalent to ~100-200 GPUs) run over a few weeks, which is a relatively modest amount of compute in the context of most large state-of-the-art models used in machine learning today."
So it wasn't out of reach for academia, pharmaceuticals, or others with a bit of resources.
These years of research involved trying many different architectures, many of which received as much or more compute time than the final system.
The price of training the final architecture is meaningless. Researching and training AlphaGo was expensive but it enabled the ideas and development of AlphaZero which is more computationally tractable.
To have any chance, an academic team would need the same compute resources as what the DeepMind protein folding team used during the whole development of the architecture during the last few years, not only the resources used to train the final system. And I bet this funding is not available to most if not all academic teams.
That said I think the academic and pharma communities had engineered themselves into a corner and weren't going to see huge gains (even thogh they are exploring similar ideas) for a number of banal reasons.
What I meant to focus on was that I think DeepMind has less of a pure money/scale advantage in this area than in some others. In something like Go or Atari game-playing, there are many academic groups researching similar things, but their resources are laughably small compared to what DeepMind threw at it. So you might argue that they got good results there in part because they directed 1000x the personnel and compute at the problem compared to what any academic group could afford. In biomed though, their peers in academia and industry are also pretty well-funded.
Here's an exemplar of how I think it evolved well in a cloud world: https://gnomad.broadinstitute.org/
that project adopts many concepts from google and others and greatly improved our analytic capabilities for large-scale genomics.
The research is the giant shoulders you stand on, the compute cost is the price of the tool you need to do the present-day work.
Both are relevant but the shoulder’s of giants are generally more accessible, particularly if we’re talking about published research and not proprietary tech.
A competing team is not starting from the same place the DeepMind team started at 5 or 10 years ago.
> I don’t think we would do ourselves a service by not recognizing that what just happened presents a serious indictment of academic science. There are dozens of academic groups, with researchers likely numbering in the (low) hundreds, working on protein structure prediction. We have been working on this problem for decades, with vast expertise built up on both sides of the Atlantic and Pacific, and not insignificant computational resources when measured collectively. For DeepMind’s group of ~10 researchers, with primarily (but certainly not exclusively) ML expertise, to so thoroughly route everyone surely demonstrates the structural inefficiency of academic science. This is not Go, which had a handful of researchers working on the problem, and which had no direct applications beyond the core problem itself. Protein folding is a central problem of biochemistry, with profound implications for the biological and chemical sciences. How can a problem of such vital importance be so badly neglected?
In short, academia got utterly schooled by a small group at Google spending a relatively small dollar amount on compute, using techniques that in hindsight are fairly described as "simplistic". There's no way around it.
If you were trying to get across the Atlantic, this would be like getting upset at a group of bridgebuilders for trying to solve the problem by building a bridge across instead of by inventing the airplane. The approaches are that different.
>The approaches are that different.
I'm not sure if that analogy applies here. DeepMind wasn't the first group tackling structure prediction with machine learning. Their success lies in the innovations that they implemented (predicting interresidue distances as opposed to contacts, for example).
Meaningless in historical terms, but meaningful in future terms. It's meaningless how long the training took because there were countless resources spent to get to that point. It's meaningful in the future, because we know that training times are fairly short, and iteration can be done fairly quickly.
Much like political dictators, they can be exceedingly efficient and have resources (and authority) to do things in spite of opposing interests.
People who faced with the narrative that countries have a monopoly on a number of aspects of life find monopolies are not a BAD THING(tm), but that they are bad for a consumer market - as a monopoly eventually blockades aspects of the market.
Big companies can suck up all the air in the room by monopolizing talent and making it harder for startups to pay the kinds of salaries needed for top tier AI research. Xerox PARC came up with all kinds of groundbreaking inventions that were never commercialized (by them). For every invention that comes out of a big company, it's worth thinking about whether it might have actually come out faster if it was borne of competition instead of a side project. Or in the grand scheme of things, if corporate taxes were higher and the money was given to a university research lab.
I think the best results may come from the middle ground. Smaller/medium companies are so worried about staying afloat or hitting their quarterly earnings that they have trouble making long term investments. Large companies are diverse and profitable enough that they can afford to blow money on things that might not pan out, but they don't have the same drive -- and in fact have some pressure to avoid being "too" innovative because it could cannibalize their existing products.
See https://ilr.law.uiowa.edu/print/volume-100-issue-5/all-i-rea...
Most people just show up to vote once every 4 years (or less) and make their decision based on the party affiliation or the wedge issue du jour, and the rest of the time pretty much ignore what's going on or don't have the power to do anything about it, which gives a lot of leeway for special interests to slide things in under the radar.
I feel like that is not too far from saying it makes one reconsider communism because good things can happen with authoritarian control.
Excited to see how follow on use of these models, by many more teams, researchers, and companies plays out over the next two decades.
This is a foundational advance!
This was a/the "holy grail" problem of molecular biology, long thought to be an automatic Nobel. It's somewhat unfair to characterise developments prior to this as insignificant. In fact by the time I was working on it, that "automatic Nobel" was no longer assumed, because the field had made quite a bit of progress, in many tiny steps by many different groups, and the assumption was it would continue in this slog until reaching some state of sufficiency for practical applications without ever seeing the sort of singular achievement that would be worthy of praise and prize.
Far more went into this breakthrough, obviously, than those TPU-hours: the development of those TPUs, for example, and assembling a team that can make use of them. The protein folding problem requires very little knowledge of biology or physics to understand and was always pre-destined for some outsider to sweep. Indeed, there was game that allowed people to solve structures by intuition alone, and, IIRC, some 13-year old Mexican kid cleaned everyone's clock some years back.
Why didn't some research group do this first? Most of them just don't have the budget. We were five people, total, IIRC, and felt pretty rich because we were computer-people getting the same budget for materials as everyone at our institution, which was all wetlab, otherwise. So I was a student being paid $20/h but with a $50,000/p.a. hardware budget. How many false start does it take before you do that run with 128TPUs "for a few weeks" that works? If you blow your budget on one gigantic Google invoice, what's going to happen to you when it doesn't pan out, and the whole institute laughs at you? Etc...
There are quite a few rather good things this problem has inspired over the years, though. Among them is CASP itself: the idea of instituting a yearly competition that gives unequivocal feedback on the state of the field and every group working on it is rather rare, I believe, and it's been successful. Indeed, it would seem that CASP was necessary to attract outside groups like Deepmind, i. e. deep-pocketed industry groups striving to prove themselves on a clearly defined problem. Chess, Jeopardy, CASP: maybe it would be worthwhile to explore not <solving x>, but <stating X as a problem that attracts Google/IBM/etc.-scale money> as a superior strategy in some cases.
There was also folding@home, pioneering the distributed-donated-computing model, and the aforementioned gamification of the problem, and hundreds of the most intricate, custom-tailed, more-or-less insane ideas people devoted months and/or careers and/or careers of their most promising post-docs to that didn't pan out.
Like cellular automata. They don't work for this, trust me. (Great hit for interactive poster sessions, though)
This is a big issue that most people miss. Having easy access to vast computational power makes such a difference for experimentation.
How much does hiring a deepmind-like team cost though? (massively more than the TPU resources?)
Still within reach of pharmaceutical industry I guess, but maybe not so easy for academia.
And they had income around 100 million in 2019 but it's all against Google, so looks like a 2 billion +/- 0.5 operation so far, and who knows if they pay for compute.
Other articles place the runrate at 500 million per year in 2019.
Which means 500 million * 6 years = 3 bn + 0.5 purchase price. = 3.5 bn. So somewhere in the 2.5 - 3.5 billion range its seems likely as total cost so far.
Nevertheless doesn't seem out of reach for a multinational.
Remember, we are looking in hindsight that it seemingly paid off. A few years ago, this was just an educated bet; only the richest companies with money to burn (from selling ads) would be willing to take on that kind of a risk.
There is barely any multinational that has the freedom Google had of planning to spend 3.5B with no ROI. Their shareholders would sue and vote the managers out.
It also looks like they came up with a brand new jiggling algorithm which is probably just V1 now, this really changes things in a significant way!
While substantial, the resources here are well within reach of many labs, research institutes, and organizations. For this result this big, I'd guess we'll have 2-6 additional implementations in the next 18 months. The problem has been 'open' for 40+ years, so that's lightening fast!
Ex: Summit has 27,648 V100 GPUs (and those V100s have Tensor units). If you're saying that only 200 GPUs are needed to replicate the experiment, that doesn't even use up 1% of Summit's available utilization.
It turned out only to be one more year instead of four (depending on whether getting to the 90~ range is "solved".
I'm curious to see if AlphaFold can do even better the next two years.
Those last mile percentages always tend to be small anyway.
This seems premature. Even though it does very well on average, there may be some areas where it struggles, and those areas may turn out to be important.
https://moalquraishi.wordpress.com/2018/12/09/alphafold-casp...
Now that the problem of static protein structure prediction has been solved (prediction errors are below the threshold that is considered acceptable in experimental measurements), we can confidently answer AlQuraishi's question:
Protein Folding just had its "ImageNet moment."
In hindsight, AlphaFold v1 represented for protein structure prediction in 2018 what AlexNet represented for visual recognition in 2012.