My question are as follows:
1. Does AlphaFold ever converge on structures in this fashion at all? As in, is the heuristic able to identify partial folds, or, is foldability somehow encoded either through evolutionary history?
2. If it does, are the structures that it identifies truly compact (measurable through sedimentation centrifugation) and solvable using conventional techniques? I suspect if the first than definitely the second because of sample bias.
3. If it does identify compact structures, are they novel sequences or sequences homologous to existing structures? If the former, then AlphaFold is truly extrapolating. If the latter, then it's only interpolating.
It would be super cool if AlphaFold discovered a "new fold" (for those who are unfamiliar with the jargon, a fold is a family of homologous sequences that create defined structure, novelty is gauged by homology, not the end structure; you can have the same structure created by wildly different sequences). It would be even cooler if AlphaFold discovered a new motif (besides alpha helix, beta sheet, and the known turns/chains). But I think that is rather unlikely.
4. If you change your bag distribution by using a primitive aa set (delete histidines, tryptophans, enrich for leucines and serines) how do your search dynamics change?
As someone who used to work in adjacent field, I have a question that I hope you can help me with. DL usually requires a large volume of data, but I don't know whether there is a huge pile of experimentally determined protein structures for training. Did DeepMind find a clever method to get around data-size issue or there is really a lot of known protein structures? I skimmed one of their earlier papers and it seems training data was in tens of thousands. I am surprised that is enough for training DL structure prediction.
Also I am not in ML research, but IIRC the spooky/counterintuitive nature of high-dimensional gradient-based search is that you can get better results from less data by increasing the number of parameters, as long as you have a sane set of regularization techniques (or have we even ditched those too?).
Maybe. In the context of natural language at least, Transformers require less and less data to reach the same result as you increase the number of parameters. No regularization needed. See Figure 2 in the paper Scaling Laws for Neural Language Models (2001.08361).
It's quite odd. Who knows if that will hold for other domains, like protein folding. It may very well be the case though, since AFAIK DeepMind's folding model used attention to reach these landmark results.
It's true you don't need regularization if you've never seen the same data twice, but that's a similar regularization to early stopping. You'd expect the larger number of parameters would make the training error drop faster with fewer tokens as well due to improved optimizability. But rather than "larger models need less data", I'd say the take away is more "larger models need fewer steps to optimize training error". None of the models get "good" until they've seen a number of tokens similar to the number of parameters.
Unless you've run many epochs on small data sets and seen the same results, in which case that's pretty weird/cool.
It is interesting that larger models with regularization (early stopping) seems to work better than than training smaller models to convergence though.
But they added a ton of highly domain specific features. And since it's essentially a natural phenomenon, I would expect the signal to be relatively strong.
I recommend the video, or any video from Yannic:
https://youtu.be/B9PL__gVxLIPretty much everything in the post is true, the title isn't clickbait, and I didn't see any fallacy. It's only "goalpost moving" if you confused the location of the goalposts in the first place. For the record: I have worked on protein folding, I did a PhD in a closely related area, and I work in computational structural biology professionally now.
The OP is saying that the general problem isn't solved, and provided (IMO factual, true) explanations why there's still a long way to go. Arguing that it's OK for some uses (which is what you're doing) is true, but not a rebuttal.
Maybe it's possible to use this method to come up with models "good enough" for rigid-body docking, or to get the overall idea of a fold. But it's also true that these structures aren't going to be good enough for any kind of drug discovery, which is what TFA says.
It is my judgement that if you presented these results in 2003 anyone would have said "the protein folding problem has been solved". Nobody then expected "the solution to protein folding" to solve every single protein, and it's silly to now. If you're complaining about the 33% error rate, you don't know enough about ML to realize that that gap is going to get closed, and rapidly.
My argument is that it is "good enough for the uses that are in the domain of protein folding". There are a lot of things that are adjacent to protein folding (enzyme pocket analysis .e.g), but they never have been considered to protein folding per se.
"It is my judgement that if you presented these results in 2003 anyone would have said "the protein folding problem has been solved"."
Your judgment is your own. At the very least, your use of the word "anyone" should give you pause.
"Nobody then expected "the solution to protein folding" to solve every single protein, and it's silly to now."
Not the argument I'm making.
"If you're complaining about the 33% error rate, you don't know enough about ML to realize that that gap is going to get closed, and rapidly."
Not the argument I'm making.
"My argument is that it is "good enough for the uses that are in the domain of protein folding"."
I have no idea what that means. Uses "in the domain of protein folding" are equivalent to uses "in the domain of structural biology". So obviously, it depends on the quality of the structure being generated.
"There are a lot of things that are adjacent to protein folding (enzyme pocket analysis .e.g), but they never have been considered to protein folding per se."
You seem to have some internal dialogue where there is a field called "protein folding", where there is an unambiguous threshold for success that has now been crossed. There is no such field, and there is no such threshold.
Are these structures good enough to do some kinds of things, like structural genomics or docking? I dunno...maybe? Are they good enough to do structure-based drug design? Not from what I've seen. TFA seems to be saying the same thing, and your comment was that TFA is just wildly wrong, which seemed...unfair, at best.
This seems to be the core of the article's arguments. Is this ratio high enough to claim the problem is "solved"?
If it was possible to accelerate drug discovery by looking at computed protein structures rather than protein structures established experimentally, it has now become plausible to do so. It has crossed a tipping point.
Since I'm not aware that there has been any drug discovery that has strongly depended on computational protein folding, this possible benefit remains to be seen.
[I worked in a lab which was heavily involved in CASP, the protein folding competition, but I was not myself involved]
Even though Google only won 4/5 games against Lee Sedol, that was the moment that sticks in peoples minds as when Google solved the problem of computer Go. What happened afterwards was a process of perfecting the algorithm. It seems likely the next few years will bring the same for AlphaFold.
https://twitter.com/TassosPerrakis/status/133353559400213299...
Tassos is also a professor in structural biology. Discovering structures the experimental way.
EDIT: on the other hand he also agrees that it's not the end of experimental structural biology
https://twitter.com/TassosPerrakis/status/133402467831107993...
Not to give DeepMind too much credit, but it's also entirely possible that "both are correct". By the nature of how targets are selected for CASP, year over year there is higher likelihood that "pathologically difficult" proteins are presented for the competition, for example - a protein that exhibits polymorphic structure where the technique (say Xtal vs NMR) biases the "solved" fold in a huge way. I believe CASP tries to weed out proteins that we know are polymorphic, but you can never be sure, and again, as time marches on, those are the proteins that fundamentally are harder to solve, so it's likely they will be enriched in the test pool.
An extreme example is insulin. The structure of insulin is has been solved for 60? I think years, but when it's in the environment of its receptor, it looks TOTALLY different (solved 5? years ago). Having said that, doubtful that DeepMind could ascertain that structure, since the environment is super-super different.
I think that the modestly high error rate is an indication that deepmind is mostly interpolating and has solved the broad protein folding heuristic. It probably will get better.
Right but we also can't generalize from 2 out 3, and without knowing this figure it's really hard to say how useful this is, no?
> Knowing how ML works, I'm surprised it didn't give any indication of low confidence.
This is actually a hard problem in ML, for example in NLP, many people assume a high log prob score means a high confidence but it is not true at all.
Thank you for the clarification. I wasn't aware of this, as I'm most familiar with super-basic/old NLP techniques like BOW/RNN/LSTM/GRU techniques, where log prob scores seemed to me to be roughly correlated with result quality. I'm aware the landscape has changed recently with insights about high dimensional searches...
But after training you can recalibrate the temperature of the softmax on the test set and still get meaningful confidence scores (temperature calibration). Or you can use a variation of cross entropy called Focal Loss that will leave your logits un-squashed.
When you set a threshold you improve precision at the detriment of recall. It's a tradeoff you can play with, but the score depends on it.
If it got answer right 1 time in 100, that would be amazing and you'd be foolish not to use it!
Except you have no way of knowing if the answer it gives you is one of the 66 right predictions vs one of the 33 wrong predictions. You could say it's likely correct, but not to a high enough degree of confidence that you could really trust it without verifying using the old established techniques.
Also important - verifying that a given model has a signature that matches the established techniques is far easier than using those techniques to generate the complete model from scratch.
I'm not really sure what your point is.
This isn't really how it works.
To quote the CASP competition organisers:
The organizers even worried DeepMind may have been cheating somehow. So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.”
But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”[1]
So you have experimental results, but still don't know how it folds. You aren't trying to avoid the all the experiments, just understand them.
[1] https://www.sciencemag.org/news/2020/11/game-has-changed-ai-...
No. Look a the actual competition results.
> As a practitioner, what I want to be able to do is pull a sequence that I've retrieved from DNA, drop it into a computer, and get the structure out. Intuitive insight can FOLLOW from those results.
According to the competition results, you cannot do that using AlphaFold2.
- Professor John Moult
Co-Founder and Chair of CASP, University of Maryland
Gee, the Co-Founder of the competition calls it a solution. I wonder if that might be because this is a solution to the problem as he A) Knows much more about the problem than you B) has actually personally seen the results and C) has far more context than you to base his decision to call it a solution.
So Lupas set a special challenge: a membrane protein from a species of archaea, an ancient group of microbes. For 10 years, his research team tried every trick in the book to get an x-ray crystal structure of the protein. “We couldn’t solve it.”
But AlphaFold had no trouble. It returned a detailed image of a three-part protein with two long helical arms in the middle. The model enabled Lupas and his colleagues to make sense of their x-ray data; within half an hour, they had fit their experimental results to AlphaFold’s predicted structure. “It’s almost perfect,” Lupas says. “They could not possibly have cheated on this. I don’t know how they do it.”
Sure, you want to validate thing experimentally, but that isn't different to imaging.
[1] https://www.sciencemag.org/news/2020/11/game-has-changed-ai-...
https://news.ycombinator.com/reply?id=25262458&goto=threads%...
So they bootstrapped the phases of the X-ray diffraction pattern with the phases from AlphaFold. This does mean, though, that we must be critical - it is possible that they are converging to nonsense that AlphaFold has generated. To be truly sure, you must have some sort of independent confirmation. Might still be "good enough" depending on what your application is.
>I have 15+ years of postgraduate chemistry/molecular bio/biophysics/biochemistry experience, then quit to go into tech, worked for AI hardware and now AI-driven software startups, mostly as backend
What made you quit biochemistry and enter tech and AI?
I'm not AS concerned about something like Ebola, because while being horrible, it's horribleness is self limiting because people die (relatively quickly) before they can spread it too far.
On the other hand, weaponized vCJD could have a population die off en masse years after they're infected. Since it can be transmitted via blood transfusions, it'd essentially shut down blood donations (although if everyone had it, I guess that wouldn't matter). Also just the fact that prion diseases can cause psychological symptoms make the implications different from something like a classical infectious disease.
Maybe the infection could use genetic mechanisms so that the infected pass it on to their children. Maybe this already happened thousands of years ago. X-Files Theme.
You can bin all astounding ml results into one of two categories, convolutional forms or sequential forms. The problem space does not fit into either of those so cleanly.
Yes, you can turn a molecule into a graph, and there are graph ml techniques, but how that interacts with a protein also does not have a lot of source data (real or synthetic) nor is the interaction space obviously reducible into a differentiable form. Either would be sufficient for today's breed of ML to excel at this.
Some context on myself: I have 15+ years of postgraduate chemistry/molecular bio/biophysics/biochemistry experience, then quit to go into tech, worked for AI hardware and now AI-driven software startups, mostly as backend (not implementing ML models, but I know how to do that and have done some small models for myself). I'm a pessimist about this AI innovation cycle's prospects in general AI and in particular drug discovery.
Has protein folding been solved? Yes. As a practitioner, what I want to be able to do is pull a sequence that I've retrieved from DNA, drop it into a computer, and get the structure out. Intuitive insight can FOLLOW from those results. For example: In one of my projects, I was able to look at the structure (a homolog had been solved, so I just did a dumb alignment and threading), identify that it was acting as an NPN transistor (unpublished details), fix the electron flow through the enzyme and improve the yield (https://link.springer.com/article/10.1186/1754-1611-7-17). Later, I looked at the structure of the enzyme, modified the surface charge in one particular part of it, and improved electron throughput again (https://www.mdpi.com/1422-0067/16/1/2020). This was with primitive "protein folding through homology" tools, now there's an good chance I could do these sorts of things with proteins I don't have homologous structure for.
These are the sorts of things that protein folding enables. One more thought -- I bet that DeepMind can do things like make it obvious where there are certain posttranslation sites (like FeS clusters) or make it obvious where there is a cryptic phosphorylation or glycosylation site (sequence holdover from a previous mutant that no longer has its expected functionality) because it's been buried.
Will there be corner cases? Yes. Probably deepmind will have difficulty solving the fold of amyloid fibrils. Probably deepmind will have some difficulty with super-strange post-translational modifications (think things like GFP's core fluorophore), or if you design a drug where you splice in an unnatural or D-amino acid, or oddities like that.
> we are not at the point where this AI tool can be used for drug discovery
I disagree. Sure, it probably won't be able to find a small molecule binding site. It almost certainly won't be able to design a drug that has a long-range allosteric effect (think Gleevec's super strange mechanism of action). But, deepmind WILL be able to help design biologics that, for example, can interact with bump-hole mutations.
There was never an expectation that "protein folding" solves every problem in the drug discovery pipeline. That's out of scope for the basic problem.
As for this:
> AI methods rely on learning the rules of protein folding from existing protein structures.
Come on. There is no method that doesn't rely on learning the rules of protein folding from existing structures. Even de novo MD-modeling has tweaking fudge factors (we could call it "dark biochemical fields" -- think: what is the expected dielectric constant around a tyrosine residue?? No way we're calculating that from the schroedinger equation) that are empirically derived to get your results.