News for hackers. Stuff that matters.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by treis on Monday November 30, 2020 @11:55AM (#25255874)
That is an impressive improvement, but I think you've missed the most important point:

>a score of around 90 GDT is informally considered to be competitive with results obtained from experimental methods

So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mabbo on Monday November 30, 2020 @03:28PM (#25258542)
Then we get the really fun question: if the experimentally determined structure is only 90% accurate, can machine learning actually reach 100%? Can you learn exact truth from inexact examples?

Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works. If so, can we somehow extract that truth, or is it truly a black box that does the thing we want?

I'm reminded of a sci-fi book I read long ago in which humans are discussing the fact that the science they are utilizing is beyond the scope of a human mind to comprehend- only the AIs can intuitively deal with 12-dimensional manifolds (or something to that extent). Maybe we've reached the doorstep of that future.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by carlmr on Monday November 30, 2020 @04:45PM (#25259431)
If you have an experimental error that is somewhat normally distributed around the mean, the the AI should, with enough examples, learn what the rules are that are closest to the mean. Because it will minimize the sum of errors.

So i do think the results could be more accurate than measurement.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by willj on Monday November 30, 2020 @09:19PM (#25261555)
I don’t think we can assume the errors are normally distributed. It’s possible researchers are biased in a particular “direction”, away from 0 on all dimensions of this problem.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by bollu on Tuesday December 01, 2020 @01:13AM (#25262867)
That's fine. It's still a normal distribution with a different mean. The Gaussian is characterized by having only the first two moments: mean and variance.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by hoseja on Tuesday December 01, 2020 @05:02AM (#25263924)
There is no 100%. Proteins are flexible. Curious how this deals with that even more fiendishly difficult fact.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by root_axis on Tuesday December 01, 2020 @12:18AM (#25262579)
> Which gets into the concept of whether the ML model has actually learned some deeper conceptual ideas than we have, some deeper truth about how this works.

Well I think that the results speak for themselves; ultimately the question you raise is one of semantics. ML models don't think in terms of "conceptual ideas" like humans do, these models simply perform at such a massive statistical scale that they can identify patterns far beyond any human conception. Clearly, the model embodies some verifiably reliable information about the way the world works, but this is "just" a trick of statistics not anything resembling actual "understanding" in the way the word is typically used when referring to human understanding.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by asdfasgasdgasdg on Monday November 30, 2020 @12:13PM (#25256128)
I have a related question about this. If experimental methods produce results around a score of 90, what is the baseline we are comparing the DeepMind results against? If the experimental error is equal to the observed DeepMind error, how can we say which one is actually more erroneous?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mrDmrTmrJ on Monday November 30, 2020 @12:38PM (#25256448)
Excellent question. At somepoint, I think the only answer is, "have a bunch of different people run a bunch of experiments on the same protein."

The threshold for "real" in particle physics is +5 sigma. Which takes a lot of data.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dnautics on Monday November 30, 2020 @11:46PM (#25262411)
you really can't compare stats like that. Those are independent, uncorrelated measurements. When you take RMSD measurements on a molecule they are not independent (for example, atoms near the core are less likely to be "inaccurate").
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by radioactivist on Monday November 30, 2020 @12:54PM (#25256671)
I think it's that a score of >90 means the result is within the error bars of whatever particular experiment was chosen to be the "reference".
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by IfOnlyYouKnew on Monday November 30, 2020 @03:29PM (#25258547)
The "experiments" here use X-Ray Crystallography. Like most methods of measuring anything, we have a pretty good idea of its accuracy under various conditions.

Think of it like satellite imagery of a tree: A score of zero would be a single green-ish pixel, while a score of 100 would show each leaf within the range it naturally moves in due to wind etc. (proteins tend to wiggle quite a bit under natural conditions, as well)

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by 0-_-0 on Monday November 30, 2020 @12:26PM (#25256293)
That's a damn good question, it looks like we don't know how much above 90 AlphaFold is.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by cpeterso on Monday November 30, 2020 @12:39PM (#25256456)
And is it even meaningful for DeepMind to score better than experimental results? How are DeepMind’s results scored then?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by marcosdumay on Monday November 30, 2020 @12:46PM (#25256561)
Finding the energy of each configuration should be much easier than finding the lowest-energy configuration. Can that be calculated ab-initio or it is still too expensive?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by crispycrafter2 on Monday November 30, 2020 @02:57PM (#25258153)
The problem with ab-initio methods in this context is the sheer number of non-covalent interactions present in these large proteins. A simple protein would require a hybrid quantum mechanic/molecular mechanics simulation to even approximate the vibrational energy required to validate equilibrium.

These proteins are so massive that we often use Daltons [1] as an averaged measure of molecular weight.

Conceptually one of the most promising applications of quantum computing is theoretical chemistry, and we are only now starting to make progress in this avenue [2]. I anticipate it would require quantum computing to explicitly optimise large folded proteins.

1. https://en.m.wikipedia.org/wiki/Dalton_(unit) 2. https://arxiv.org/abs/2004.04174

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by timr on Monday November 30, 2020 @02:02PM (#25257499)
"So DeepMind is to the point where it's a question of whether their generated model or the experimentally determined structure is closest to the actual physical structure."

While this is an accomplishment, nobody is going to be confusing these models for structures produced experimentally. The CASP metric is for backbone atoms. To have a useful model of protein structure, you really need to have the positions of the protein side-chain atoms modeled correctly. Experimental methods will do that, but this method, as I understand it, does not.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by jey on Monday November 30, 2020 @02:16PM (#25257661)
So it's a really good start, but nobody is going to be throwing these structures into molecular docking simulations for drug discovery or etc just yet. But hopefully those details can be worked out soon enough.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by timr on Monday November 30, 2020 @03:32PM (#25258587)
Yeah, there's a huge difference between a 1Å all-atom RMSD structure, and a 1Å backbone RMSD structure. The non-backbone atoms in a protein make up most of the mass and volume. When structural biologists talk about RMSD, this is what they mean.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mFixman on Monday November 30, 2020 @05:42PM (#25259993)
I don't have a background in biology, and that quote confused me.

What's an experimental method for protein folding and why is it so good? Are they talking about creating an actual, physical protein in a lab and observing how it folds?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by flobosg on Monday November 30, 2020 @05:48PM (#25260041)
> Are they talking about creating an actual, physical protein in a lab and observing how it folds?

Exactly. Researches purify the folded protein and then use methods such as X-ray crystallography, nuclear magnetic resonance, and cryo-electron microscopy to determine its three-dimensional atomic structure.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by selestify on Monday November 30, 2020 @06:36PM (#25260464)
If even the experimental approach is only 90% accurate, how do they know which 90% is accurate?
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by rsfern on Monday November 30, 2020 @09:35PM (#25261655)
I’m not a protein crystallographer, but here’s my generalist take.

We understand the physics of e.g. X-ray diffraction pretty well, so we can fit pretty decent forward models for the x-ray data given a proposed structure. The hardest task here is getting a good enough guess at the structure to optimize the physical model, and it’s my impression that people use an iterative model refinement workflow. At least that’s how it’s done in condensed matter materials.

There are many sources of experimental uncertainty, like the non-ideal nature of the x-ray source and optics, and the fact that the atoms in the protein are not static but have some thermal fluctuations. so at the end of the refinement you still have some uncertainty on your model parameters (the interatomic distances for proteins I guess), but if you are careful you can calibrate these uncertainties pretty well.

This paper looks like a really good detailed discussion: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC4080831/

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dnautics on Monday November 30, 2020 @11:54PM (#25262458)
X-ray diffraction is pretty nutty, too. You're taking the diffraction pattern, which is the fourier transform of the electron density. Fourier transform results are complex-valued data. Unfortunately, we don't really have X-Ray lasers, so you can only get the intensities and not the phases of those diffraction spots. Since mother science hates us, it of course the case that, in a fourier transform "more information" is contained in the phases than in the intensities.

So you "make guesses at what the phases are", the best choice is to bootstrapping these phases measured with another technique (you can introduce crystal defects that do allow you to guess at what the phases are).

Less scrupulous is to use a computer generated model, like fitting another protein "that you guess is related", then you model the electron density, take the phases of that.

In any case you take these "phase" guesses, and then apply it to your intensities, re-run the fourier transform, refine your electron densities, twiddle the location where you think the atoms, are, then repeat with your new model. This process repeats until you converge on a structure that you're happy with.

Now alarm bells should be screaming in your head right now: Yes, it's entirely possible to converge on a wrong structure, especially if you're a young up-and-comer professor seeking tenure that has no ethical problems with "suggesting" their grad students to sleep in the lab and work 100 hour weeks and willing to do slipshod work to get you tenure: https://www.sciencedirect.com/science/article/pii/S002228360...

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by selestify on Tuesday December 01, 2020 @06:56PM (#25271688)
> Yes, it's entirely possible to converge on a wrong structure

I guess my question is, how do you know if you’ve converged on the right structure or not? Is there a different experiment you could do?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dnautics on Wednesday December 02, 2020 @10:57AM (#25277537)
Best is an orthogonal process (like NMR). Cryo-EM is getting better too so maybe that will start to be viable. Sometimes that's not possible, but you can use secondary evidence: "we know these three amino acids are important band hey look they touch in our model".
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by mactrey on Monday November 30, 2020 @12:57PM (#25256701)
I'm not a biologist but I'm not sure that follows. It could be that the experimentally-derived structure is 100% accurate to the actual physical structure but getting 90% of your predicted residues to match that is enough to get an accurate prediction of protein behavior and hence "competitive."
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by SubiculumCode on Monday November 30, 2020 @12:45PM (#25256521)
Something like this comes up in assessing the accuracy of automated segmentation results of brain regions e.g. the hippocampus. Human-machine reliability is approaching the human to human reliability, so it becomes harder to improve the automated methods.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by contravariant on Monday November 30, 2020 @04:04PM (#25258951)
Of course this may no longer be the case for methods solely trained to optimize that particular metric.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by beowulfey on Monday November 30, 2020 @12:55PM (#25256684)
I don’t think you can say DeepMind could ever be more accurate to the true physical structure since it was built on the same experimental structures that it is being compared to. The limit of accuracy is the experimental data. However, I think we can say that a DeepMind prediction could at least be as good as a new experimental structure.
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dwiel on Monday November 30, 2020 @01:37PM (#25257241)
This seems like an obvious assumption to make, but it isnt always true. It is easier to see why if you are measuring a single value multiple times in order to get a more accurate estimate of the true value. In that case your "model" is simply the mean of all measurements made and can exceed the accuracy of a single measurement.

In this case, the model is predicting values of multiple structures, but patterns could still theoretically be found which allow for predictions beyond the accuracy of a single measurement.

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by dekhn on Monday November 30, 2020 @05:28PM (#25259864)
DM is merging several experimental data: known x-ray structures, and evolutionary data. The experimental method (xray) doesn't take advantage of the evolutionary data. And it also doesn't model the underlying protein behavior accurately (xray basically assumes a single static model with atoms fluctuating in little gaussian "puffs" around the atomic centers, but that's not how most proteins behave).
Re:AlphaFold: a solution to a 50-year-old grand challenge in... by robocat on Monday November 30, 2020 @01:57PM (#25257455)
But DeepMind could be used to find errors in the training set.

Let’s say you have 100000 proteins in the training set. Now remove #1 and train on 99999, and then check that it still predicts the same protein result for #1 as the experimental result.

Or remove from training whole sets of proteins by particular teams to find systematic errors made by teams?

Re:AlphaFold: a solution to a 50-year-old grand challenge in... by FrojoS on Monday November 30, 2020 @01:44PM (#25257323)
Is that true? I thought fundamentally, the simulation tries to find the state of lowest energy, which is defined by physics. So, your result can be better than the data set used for training.