News for hackers. Stuff that matters.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mabbo on Monday November 30, 2020 @11:19AM (#25255349)
Sometimes announcements like this are a bit over-the-top. But what really, to me, cements the 'big-deal' of this is the "Median Free-Modelling Accuracy" graph half way down the page.

Scores of 30-45 for 15 years. Now scores of 87-92.

This isn't a minor improvement, it's a leap forward.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by treis on Monday November 30, 2020 @11:55AM (#25255874)
That is an impressive improvement, but I think you've missed the most important point:

>a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods

So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mabbo on Monday November 30, 2020 @03:28PM (#25258542)
Then we get the really fun question: if the experimentally determined structure is only 90% accurate, can machine learning actually reach 100%? Can you learn exact truth from inexact examples?

Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works. If so, can we somehow extract that truth, or is it truly a black box that does the thing we want?

I'm reminded of a sci-fi book I read long ago in which humans are discussing the fact that the science they are utilizing is beyond the scope of a human mind to comprehend- only the AIs can intuitively deal with 12-dimensional manifolds (or something to that extent). Maybe we've reached the doorstep of that future.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by carlmr on Monday November 30, 2020 @04:45PM (#25259431)
If you have an experimental error that is somewhat normally distributed around the mean, the the AI should, with enough examples, learn what the rules are that are closest to the mean. Because it will minimize the sum of errors.

So i do think the results could be more accurate than measurement.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by willj on Monday November 30, 2020 @09:19PM (#25261555)
I don’t think we can assume the errors are normally distributed. It’s possible researchers are biased in a particular “direction”, away from 0 on all dimensions of this problem.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by bollu on Tuesday December 01, 2020 @01:13AM (#25262867)
That's fine. It's still a normal distribution with a different mean. The Gaussian is characterized by having only the first two moments: mean and variance.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by hoseja on Tuesday December 01, 2020 @05:02AM (#25263924)
There is no 100%. Proteins are flexible. Curious how this deals with that even more fiendishly difficult fact.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by root_axis on Tuesday December 01, 2020 @12:18AM (#25262579)
> Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works.

Well I think that the results speak for themselves; ultimately the question you raise is one of semantics. ML models don't think in terms of "conceptual ideas" like humans do, these models simply perform at such a massive statistical scale that they can identify patterns far beyond any human conception. Clearly, the model embodies some verifiably reliable information about the way the world works, but this is "just" a trick of statistics not anything resembling actual "understanding" in the way the word is typically used when referring to human understanding.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by asdfasgasdgasdg on Monday November 30, 2020 @12:13PM (#25256128)
I have a related question about this. If experimental methods produce results around a score of 90, what is the baseline we are comparing the DeepMind results against? If the experimental error is equal to the observed DeepMind error, how can we say which one is actually more erroneous?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mrDmrTmrJ on Monday November 30, 2020 @12:38PM (#25256448)
Excellent question. At somepoint, I think the only answer is, "have a bunch of different people run a bunch of experiments on the same protein."

The threshold for "real" in particle physics is +5 sigma. Which takes a lot of data.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dnautics on Monday November 30, 2020 @11:46PM (#25262411)
you really can't compare stats like that. Those are independent, uncorrelated measurements. When you take RMSD measurements on a molecule they are not independent (for example, atoms near the core are less likely to be "inaccurate").
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by radioactivist on Monday November 30, 2020 @12:54PM (#25256671)
I think it's that a score of >90 means the result is within the error bars of whatever particular experiment was chosen to be the "reference".
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by IfOnlyYouKnew on Monday November 30, 2020 @03:29PM (#25258547)
The "experiments" here use X-Ray Crystallography. Like most methods of measuring anything, we have a pretty good idea of its accuracy under various conditions.

Think of it like satellite imagery of a tree: A score of zero would be a single green-ish pixel, while a score of 100 would show each leaf within the range it naturally moves in due to wind etc. (proteins tend to wiggle quite a bit under natural conditions, as well)

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by 0-_-0 on Monday November 30, 2020 @12:26PM (#25256293)
That's a damn good question, it looks like we don't know how much above 90 AlphaFold is.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by cpeterso on Monday November 30, 2020 @12:39PM (#25256456)
And is it even meaningful for DeepMind to score better than experimental results? How are DeepMind’s results scored then?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by marcosdumay on Monday November 30, 2020 @12:46PM (#25256561)
Finding the energy of each configuration should be much easier than finding the lowest-energy configuration. Can that be calculated ab-initio or it is still too expensive?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by crispycrafter2 on Monday November 30, 2020 @02:57PM (#25258153)
The problem with ab-initio methods in this context is the sheer number of non-covalent interactions present in these large proteins. A simple protein would require a hybrid quantum mechanic/molecular mechanics simulation to even approximate the vibrational energy required to validate equilibrium.

These proteins are so massive that we often use Daltons [1] as an averaged measure of molecular weight.

Conceptually one of the most promising applications of quantum computing is theoretical chemistry, and we are only now starting to make progress in this avenue [2]. I anticipate it would require quantum computing to explicitly optimise large folded proteins.

1. https://en.m.wikipedia.org/wiki/Dalton_(unit) 2. https://arxiv.org/abs/2004.04174

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by timr on Monday November 30, 2020 @02:02PM (#25257499)
"So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure."

While this is an accomplishment, nobody is going to be confusing these models for structures produced experimentally. The CASP metric is for backbone atoms. To have a useful model of protein structure, you really need to have the positions of the protein side-chain atoms modeled correctly. Experimental methods will do that, but this method, as I understand it, does not.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by jey on Monday November 30, 2020 @02:16PM (#25257661)
So it's a really good start, but nobody is going to be throwing these structures into molecular docking simulations for drug discovery or etc just yet. But hopefully those details can be worked out soon enough.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by timr on Monday November 30, 2020 @03:32PM (#25258587)
Yeah, there's a huge difference between a 1Å all-atom RMSD structure, and a 1Å backbone RMSD structure. The non-backbone atoms in a protein make up most of the mass and volume. When structural biologists talk about RMSD, this is what they mean.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mFixman on Monday November 30, 2020 @05:42PM (#25259993)
I don't have a background in biology, and that quote confused me.

What's an experimental method for protein folding and why is it so good? Are they talking about creating an actual, physical protein in a lab and observing how it folds?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by flobosg on Monday November 30, 2020 @05:48PM (#25260041)
> Are they talking about creating an actual, physical protein in a lab and observing how it folds?

Exactly. Researches purify the folded protein and then use methods such as X-ray crystallography, nuclear magnetic resonance, and cryo-electron microscopy to determine its three-dimensional atomic structure.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by selestify on Monday November 30, 2020 @06:36PM (#25260464)
If even the experimental approach is only 90% accurate, how do they know which 90% is accurate?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by rsfern on Monday November 30, 2020 @09:35PM (#25261655)
I’m not a protein crystallographer, but here’s my generalist take.

We understand the physics of e.g. X-ray diffraction pretty well, so we can fit pretty decent forward models for the x-ray data given a proposed structure. The hardest task here is getting a good enough guess at the structure to optimize the physical model, and it’s my impression that people use an iterative model refinement workflow. At least that’s how it’s done in condensed matter materials.

There are many sources of experimental uncertainty, like the non-ideal nature of the x-ray source and optics, and the fact that the atoms in the protein are not static but have some thermal fluctuations. so at the end of the refinement you still have some uncertainty on your model parameters (the interatomic distances for proteins I guess), but if you are careful you can calibrate these uncertainties pretty well.

This paper looks like a really good detailed discussion: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4080831/

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dnautics on Monday November 30, 2020 @11:54PM (#25262458)
X-ray diffraction is pretty nutty, too. You're taking the diffraction pattern, which is the fourier transform of the electron density. Fourier transform results are complex-valued data. Unfortunately, we don't really have X-Ray lasers, so you can only get the intensities and not the phases of those diffraction spots. Since mother science hates us, it of course the case that, in a fourier transform "more information" is contained in the phases than in the intensities.

So you "make guesses at what the phases are", the best choice is to bootstrapping these phases measured with another technique (you can introduce crystal defects that do allow you to guess at what the phases are).

Less scrupulous is to use a computer generated model, like fitting another protein "that you guess is related", then you model the electron density, take the phases of that.

In any case you take these "phase" guesses, and then apply it to your intensities, re-run the fourier transform, refine your electron densities, twiddle the location where you think the atoms, are, then repeat with your new model. This process repeats until you converge on a structure that you're happy with.

Now alarm bells should be screaming in your head right now: Yes, it's entirely possible to converge on a wrong structure, especially if you're a young up-and-comer professor seeking tenure that has no ethical problems with "suggesting" their grad students to sleep in the lab and work 100 hour weeks and willing to do slipshod work to get you tenure: https://www.sciencedirect.com/science/article/pii/S002228360...

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by selestify on Tuesday December 01, 2020 @06:56PM (#25271688)
> Yes, it's entirely possible to converge on a wrong structure

I guess my question is, how do you know if you’ve converged on the right structure or not? Is there a different experiment you could do?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dnautics on Wednesday December 02, 2020 @10:57AM (#25277537)
Best is an orthogonal process (like NMR). Cryo-EM is getting better too so maybe that will start to be viable. Sometimes that's not possible, but you can use secondary evidence: "we know these three amino acids are important band hey look they touch in our model".
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mactrey on Monday November 30, 2020 @12:57PM (#25256701)
I'm not a biologist but I'm not sure that follows. It could be that the experimentally-derived structure is 100% accurate to the actual physical structure but getting 90% of your predicted residues to match that is enough to get an accurate prediction of protein behavior and hence "competitive."
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by SubiculumCode on Monday November 30, 2020 @12:45PM (#25256521)
Something like this comes up in assessing the accuracy of automated segmentation results of brain regions e.g. the hippocampus. Human-machine reliability is approaching the human to human reliability, so it becomes harder to improve the automated methods.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by contravariant on Monday November 30, 2020 @04:04PM (#25258951)
Of course this may no longer be the case for methods solely trained to optimize that particular metric.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by beowulfey on Monday November 30, 2020 @12:55PM (#25256684)
I don’t think you can say DeepMind could ever be more accurate to the true physical structure since it was built on the same experimental structures that it is being compared to. The limit of accuracy is the experimental data. However, I think we can say that a DeepMind prediction could at least be as good as a new experimental structure.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dwiel on Monday November 30, 2020 @01:37PM (#25257241)
This seems like an obvious assumption to make, but it isnt always true. It is easier to see why if you are measuring a single value multiple times in order to get a more accurate estimate of the true value. In that case your "model" is simply the mean of all measurements made and can exceed the accuracy of a single measurement.

In this case, the model is predicting values of multiple structures, but patterns could still theoretically be found which allow for predictions beyond the accuracy of a single measurement.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dekhn on Monday November 30, 2020 @05:28PM (#25259864)
DM is merging several experimental data: known x-ray structures, and evolutionary data. The experimental method (xray) doesn't take advantage of the evolutionary data. And it also doesn't model the underlying protein behavior accurately (xray basically assumes a single static model with atoms fluctuating in little gaussian "puffs" around the atomic centers, but that's not how most proteins behave).
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by robocat on Monday November 30, 2020 @01:57PM (#25257455)
But DeepMind could be used to find errors in the training set.

Let’s say you have 100000 proteins in the training set. Now remove #1 and train on 99999, and then check that it still predicts the same protein result for #1 as the experimental result.

Or remove from training whole sets of proteins by particular teams to find systematic errors made by teams?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by FrojoS on Monday November 30, 2020 @01:44PM (#25257323)
Is that true? I thought fundamentally, the simulation tries to find the state of lowest energy, which is defined by physics. So, your result can be better than the data set used for training.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by WhompingWindows on Monday November 30, 2020 @12:11PM (#25256097)
This reminds me of AlphaGo and AlphaZero. DeepMind was able to produce a very solid model on their first attempt, at both protein folding and at Go (and Starcraft2 as well). Their second models, however, seemed to blow their first out of the water.

This bodes extremely well for the future of computational biology, I'm very excited thinking about the prospects. If we know how a protein folds, we know its shape, meaning we know which shaped/charged molecules are needed to act as suppressors/enhancers of those proteins.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by layer8 on Monday November 30, 2020 @02:29PM (#25257838)
One difference to AlphaZero though, if my understanding is correct, is that AlphaFold is trained on a predetermined data set and hence didn’t learn how “arbitrary” proteins fold in general, but just how the kinds of proteins fold for which we already know how they fold. To work more like AlphaZero, AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions. Therefore it’s conceivable that AlphaFold is biased by the existing training data and doesn’t fully generalize to all proteins we would want to apply it to. Maybe that won’t be a problem in practice, but nevertheless it makes for a significant difference from what AlphaZero was about, being solely self-trained.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by the8472 on Monday November 30, 2020 @02:45PM (#25258036)
> AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions.

Could this lead to a virtuous cycle where AlphaFold is used generate a ton of random sequences where it has low confidence, those are then screened for ease of synthesis, measured and the results used to improve the model?

Edit: nevermind, according to another comment[0] there are still plenty of real proteins without experimental data left to explore.

[0] https://news.ycombinator.com/item?id=25255601

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by cma on Tuesday December 01, 2020 @12:43AM (#25262706)
> AlphaFold would have to be able to synthesize arbitrary proteins and run the experiments on them to verify and correct its predictions.

It can verify how much it minimizes the potential energy, which may not always line up with how it would fold in the real world but is a strong indicator.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by entropicdrifter on Monday November 30, 2020 @11:42AM (#25255700)
Not to mention the fact that two years ago they took it from 45% to >60%. If they can continue improving, even with an exponential decay in rate of improvement, this is certainly a stunning example of technological disruption.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by Zenst on Monday November 30, 2020 @11:48AM (#25255779)
Even without any improvement, the amount of grunt-work the AI can pre-do and get down to a short-list - that in itself will see changes in progress speeding research up.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by kordlessagain on Monday November 30, 2020 @11:52AM (#25255835)
> and get down to a short-list

There's no reason to believe the list will contain all solutions, however.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by patagurbon on Monday November 30, 2020 @02:26PM (#25257786)
No but it will hopefully contain some. Which for many if not most problems is all that matters
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by martinpw on Monday November 30, 2020 @01:36PM (#25257229)
Why is the graph not monotonically increasing? Does the complexity of the problem to be solved increase each time? If so, does that make the relative improvement from the previous result even more impressive?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by breatheoften on Monday November 30, 2020 @01:43PM (#25257304)
That's quite interesting ... I believe the test set size is not constant year to year but rather a function of how many new structures have been experimentally discovered since the last contest?

Does seem like the contest structure could include quite a bit of risk for hiding the effect of overfitting ... I wonder if there is anything inherent about the problem that reduces that risk ...?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by FrojoS on Monday November 30, 2020 @01:50PM (#25257390)
My understanding is, that it's always 100 new structures, which is a small fraction of the total structures identified in that year.

The reason why the top score in one year, can be lower than in the previous year, is that the test (the 100 structures to guess) is always new and different, so it can end up being 'harder' than the year before. Luck will also play a small role.

Another explanation for a reduction in the top score would be, that previous winners are not re-submitted unchanged. For instance AlphaFold v1 seems to not have been submitted to the latest competition.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by breatheoften on Monday November 30, 2020 @02:47PM (#25258056)
Only 100 new structures each test cycle? That seems a very small test set size ...

Is it really possible to select 100 new structures which together are likely to represent a meaningful increase in the sample generalization versus the prior years test set ...?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by MauranKilom on Monday November 30, 2020 @03:04PM (#25258247)
Given that we only know the structure of on the order of 100k proteins, we might only get another 10k new ones per year. I guess.

Using 1% of those (presumably from the more-often-reproduced subset) for this challenge seems reasonable? Note that the structures have to remain secret up until the challenge, and presumably all those teams uncovering the structures don't want to have to wait up to 2 years every time to actually make their results public.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by breatheoften on Monday November 30, 2020 @03:19PM (#25258429)
Interesting ... plenty of opportunity then potentially for the 100 samples to have prediction similarity to the set of published discoveries (for expected or unknown reasons)?

I suppose it will take a few more years of repetition for the challenge to confirm that the problem has been been solved -- but I wonder if a new version of the contest is going to be needed as well? Maybe the model accuracy is now high enough to invert the contest to a form where models generate predictions for randomly selected unknown samples -- and experimental teams are then expected to make observations for those particular sequences over the next two years as part of their otherwise research agenda selected experimental workload?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by kxs on Tuesday December 01, 2020 @04:29AM (#25263773)
There are different categories of samples, namely FM and TBM targets. FM targets don't have any similarity to known structures. Roughly a quarter were FM targets. I think the more interesting thing to look at is the size of the multiple sequence alignments (MSAs) which is the basis of this and essentially all methods. They seem to do very well with few MSAs, which bodes well for other targets, although there are families of proteins with few MSAs.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by hobofan on Tuesday December 01, 2020 @03:55AM (#25263637)
100 structure with 100+ amino acids each, so it's not quite as bad. Part of the folding information is contained within a distance of a few amino acids, while some (the harder part and crux of the problem) is farther away.

But yeah, compared to other fields, the size of training/test sets is sometimes pretty small in ML for life sciences.