It's more of a philosophical question really, but are AlphaGo and AlphaFold "producing" knowledge? AlphaGo plays go, AlphaFold predicts protein structure. They are useful tools, but I wouldn't call what they produce knowledge. However, as you say, their underlying models have certainly captured knowledge that we, as humans, do not comprehend yet.

So I would reformulate the observation of the article as follows: we have become extremely good at building useful tools that embeds knowledge that we do not understand.

If I may attempt an analogy, it reminds of Ptolemaic tables: quite useful at predicting planetary motions with some accuracy. What they lacked in understanding, they made up with accumulated data and complexity. Maybe AlphaFold is just that: a Ptolemaic table built on top of a huge swath of data.

We will probably build a lot of these tools in the future, as we have very powerful tools to build them. But I don't think they will replace a more fundamental understanding (that is to say science), that will simply lag behind.

If someone or something produces knowledge, then yes, they are scientists and they need not be human. However it must at least be intelligible to them, and in that case it is that entity doing science.

ML systems as they exist do not produce knowledge for humans, and they do not understand anything themselves. They're mechanisms that do work. A chess computer doesn't understand chess any more than a lever understands physics, even if they can beat humans or lift heavy objects. Your TI-83 is not a mathematician despite the fact that it can calculate much quicker than you can.

If you're suggesting that at some point we may built some sort of AGI-scientist that can actually reason about something, potentially at a level we can't understand, sure. But that is not what ML is, that is science fiction, and it is not how ML systems will be used in the scientific process, they will aid as tools as they're doing right now, and that is what everyone will tell you who worked on AlphaFold.

You could say the same about humans; that we do not understand anything ourselves; that we’re mechanisms that do work. But it’s a moot point.

What I’m saying is that “reason” is an anthropocentric concept and is not a prerequisite for the production of knowledge, or even conducting science for that matter.

Reason isn't an anthropocentric concept at all. Knowledge isn't even produced. Knowledge is the subjective representation of facts, objects, skills and so forth. What's being produced by something like Alphafold is data, interpreting that data by means of reason and abstracting it into a model is what we call knowledge. The entire process is science.

And the first point you make is very important. The fact that we have neural nets in our heads that we didn't and don't really understand hasn't changed anything about what science is. We reason about the world and we do science because having general models about the world we are in is more powerful than merely collecting or correlating data. And that will be the case for any kind of intelligence.

We don't have a perfect model of reality to do science completely virtually using unsupervised ML. Science by definition is not mathematics as it is grounded first and foremost in empiricism. You can use ML to help produce science. E.g. hook up a sufficiently sophisticated "AI" system to do experiments and analyse data and generate hypothesises. But the AI itself won't obsolete the scientific method. I am surprised that the posted article is written by a science journalist, I suspect it is mostly due to her not understanding how ML is implemented rather than misunderstanding the scientific method.

> But the AI itself won't obsolete the scientific method

I wouldn't put a lot of money on that in the future given that even 3 years ago AI could prove about 58% of a set of theorems: https://mathscholar.org/2019/04/google-ai-system-proves-over...

From that to inferring then proving new theorems in the future or even inferring new theories in other fields (that can then be tested empirically if needed) I think we're not very far from what the person you replied to suggested

However, it just modeled a known algorithm for solving theorems mechanically. 3-SAT predates ML, and it was trained using it. Essentially parroting an algorithm is not science.

Call me again when AI develops a new theorem from scratch, in a concise manner

We have a tendency to reify science as something sacred or uniquely human, but the scientific method is essentially an algorithm. Like the article said, AI does not need to develop theorems in a post-theoretical mode of science. AI only needs a systematic way to make discoveries that are in the self interest of the system in which it's embedded.

Humans may need theory to perform science, but only because we're slow and have bad memories is it necessary for us to rely on faint signals like intuition, eureka moments, and pulling all-nighters in the lab. Machines don't need narratives and metaphors to represent their knowledge of the world, nor do they need reason or explanations of how and why. They just need to function.

Or it may be that the author has an understanding which you are lacking.

> We may not know how AlphaGo plays Go, but I think we can say AlphaGo knows how to play Go.

To be honest, before I can agree or disagree with that, I'd really like to know more about how AlphaGo actually plays Go. Like, how the network is structured, how an actual decision is produced, etc.

I feel, we're talking about increasingly lofty and abstract concepts while we don't have much of a clue what actually goes on on the ground.