The architecture of the system and design of the training methodology are laid out to specifically prevent direct database-esque "pattern in pattern out" failure mode.

Similar to Google deep dream, there will be contextual features and feature clusters encoded into neurons that can be explored and extracted, and those could provide insights that can be sourcing translated into "hard" science, with explicit formulae and theory allowing a fully transparent model to be created.

Like other transformer models, you can elicit the training data intact, but such scenarios are a statistically insignificant selection of the range of outputs the models are capable of producing. That doesn't mean anything with regards to accuracy of the novel output, though.

With alphafold 2 going open source, it's possible that tools and methodologies to extract hard science from transformers will be formalized quickly and in the public eye. We have an amazingly powerful new tool, and the coming decades will be fascinating.

- The thing I'd be curious about is whether or not "not formalized" would imply "not consistently generalizing", whether it would have to be train all over if given a problem similar to but identical with, the problem it solves.

The tools to do that with transformers in particular and neural networks in general are still pretty new and specific to particular models.

https://ai.googleblog.com/2015/06/inceptionism-going-deeper-...

Someone with extensive knowledge of protein folding and chemistry and any relevant domain would need to manually categorize different neurons, their contexts, and effects, then experiment with the model in a static/procedural inference mode, then parse out the overall functions. By reproducing well known simple instances and tracing the propagation of information through the network, you can validate what a model knows by mapping to current knowledge.

Using the model in this way can also lead to interesting real world experiment choices, by processing targets suggested by real life phenomena and then coordinating from known functions of the model and real life experiment. Different proteins in coronavirus, for example, would likely have a different cascade of activations in the model than the flu virus. Any shared features between the two might be captured in the activation patterns, possibly leading to new information about the diseases that would opaque to science without these models.

So an extremely large DL network would have a good chance to find and integrate ("compress") all the existing folds and sub-folds that human researchers or Rosetta missed or just hadn't the time to investigate and characterize yet (I'm not an expert on Rosetta by far btw so please expand if you are :).

I would venture to say it's a good problem fit for DL methods (as was impressively demonstrated).

Regarding your question, "does it understand something about the structure of the protein folding problem" - expanding on the above, I would say it understands enough, but it probably doesn't understand the generics of chemistry as protein and their folding is a biased subset. The output is (as far as I remember) an atom distance matrix and not atom trajectories etc. so folding dynamics is not part of the model (this is btw an important part of protein science as well).

From what I understand, deep learning, although opaque and relying on tons of data, is a bit magical : although one would say "it's just probabilities", it actually does probabilities at a level where it actually figures some things.

Plus, and that's very much a problem to me, Google does it at 100x the scale of a regular researcher. Since I just invested a year in studying data sciences, that worries me a lot : where am I suppose to work if, to produce meaningful results, you need way-too-expensive hardware...

If you are in a team that is looking at problems that justify massive hardware (in the sense that solving them will pay back the capital and environmental cost) then you will have access to said hardware.

Most (almost all) AI and Data Science teams are not working on that kind of problem though, and it's often the case that we are working on cloud infrastructures where GPU's and TPU's can be accessed on demand and $100's can be used to train pretty good models. Obviously models need to be trained many times so the crossover point from a few $100 to a few $1000 can be painful - but actually many/most engagements really only need models that cost <$100 to train.

Also many of the interesting problems that are out there can utilize transfer learning over shared large pretrained models such as Resnet or GPT-2 (I know that in the dizzying paced modern world these no longer count as large or modern but they are examples...) So for images and natural language problems we can get round the intractable demand for staggeringly expensive compute.

Imagine that you had got a degree in Aeronautical Engineering, you are watching the Apollo program and wondering how you will get a job at NASA or something similar... but there are lots of jobs at Boeing and Lockheed and Cessna and so on.

I know how you feel, but I also think stories like this may be a wake-up call for some groups to invest more in hardware. The Baker and Tsinghua University groups are not small. They can afford more than 4 GPUs.

Probably it's more about setting up a good pipeline. Once you get about 4 GPUs you need more than one machine to run them. Hopefully in the next years we'll see more open source tools to make it easy to "build your own GPU cloud".

Anyway these days I see a lot of industrial investment in middle size computing datacenters full of GPUs. Sure Google scale is not within reach but I'm sure there's room for scrapier algorithms and training methods to demonstrate feasibility and then paying a lump sum to AWS (or some ml-expert-for-cloud-training SME) for the final step.

Anyway, I thought the expensive part was data acquisition and labeling? I like the 'surrogate network' approach of learning a very-expensive-to-compute simulation or model, that doesn't need data, but the output of a costly simulation.

>Scientific logic is proving things by losslessly compressing statements to their axioms. Commonsense logic uses lossy compression, which makes it less accurate in edge cases, but also less brittle, more efficient, further reaching and more stable in most real-world situations.

EDIT: A simpler example might be helpful. We could, for example, train a network to recognize and predict orbital trajectories. Feed it either raw images or processed position-and-magnitude readings, and it outputs predicted future observations. One could ask, "does it really understand orbital mechanics, or is it merely finding an efficient compression of the solution space?"

But this question can be reduced in such a way as to made empirical by presenting the network with a challenge that requires real understanding to solve. For example, show it observations of an interstellar visitor on a hyperbolic trajectory. ALL of its training data consisted of observations of objects in elliptical orbits exhibiting periodic motion. If it is simply matching observation to its training data, it will be unable to conceive that the interstellar visitor is not also on a periodic trajectory. But on the other hand if it really understood what it was seeing then it would understand (like Kepler and Newton did) that elliptical motion requires velocities bounded by an upper limit, and if that speed is exceeded then the object will follow a hyperbolic path away from the system, never to return. It might not conceive these notions analytically the way a human would, but an equivalent generalized model of planetary motion must be encoded in the network if it is to give accurate answers to questions posed so far outside of its training data.

How you translate this into AlphaFold I'm not so certain, as I lack the domain knowledge. But a practical ramification would be the application of AlphaFold to novel protein engineering. If AlphaFold lacks "real understanding", then its quality will deteriorate when it is presented with protein sequences further and further removed from its training data, which presumably consists only of naturally evolved biological proteins. Artificial design is not as constrained as Darwinian evolution, so de novo engineered proteins are more likely to diverge from AlphaFold's training data. But if AlphaFold has an actual, generalized understanding of the problem domain, then it should remain accurate for these use cases.

For example specific points on this object are a folded proteins. I suppose then it is how well does this get encoded, does it know about "properties" of this surface, or is it more like a rough kind of point cloud because you have sampled enough and then it does some crude interpolation. But maybe that does not respect the sort of properties in this object. Maybe there are conservation laws, symmetry properties, etc which are actually important, and then not respecting that you have just produced garbage.

So I think it is important to know what kind of problem you are dealing with. Imagine a long time scale n-body problem with lots of sensitivity. Maybe in a video game it doesn't matter if there is something non physical about what it produces, as long as it looks good enough.

Maybe this interpolation is practical for its purpose.

But I think we should still be careful and question what kind of problem it is applied to perhaps. Maybe it's more like a complexity vs complicated question.

> does it know about "properties" of this surface, or is it more like a rough kind of point cloud because you have sampled enough and then it does some crude interpolation

Say that there existed some high-level property such as "conservation of energy". A "knowledge system" which learns about that property would be able to answer any questions related to it after reducing to a "conservation of energy" problem. Is the same true for NNs? The way folks talk about them, they sound like they can compress dynamically, and would therefore be able to learn and apply new high-level properties.

Also, do NNs have "rounding errors"? We have confidently learned that energy is conserved, but would NNs which never had that rule directly encoded understand conservation as "exactly zero", or "zero with probability almost 1", or "almost zero"?