The line is: building an internal world model requires interfacing with the world, not a model of it, and subsequent failing (including death and survivorship over generations) and adaptation. Plus pattern matching.

Current AI only does one of those (pattern matching, not evolution), and the prospects of simulating evolution is kind of bleak, given I don’t think we can simulate a full living cell yet from scratch? Building a world model requires life (or something that has undergone a similar evolutionary survivorship path), not something that mimics life.

You don't need to simulate a full living cell to have evolution. In fact, isn't using evolving programs a decades-old technic ?

Genetic programming models a natural process of evolution to do something useful, the same way machine learning models neurons to do something useful.

But producing something useful is a totally different thing from producing resilience in physical reality. That takes a world model, and I guess my suspicion is that an entity can’t build a world model without a long history of surviving in that world.

Put another way, you can never replicate what it’s like to burn your hand on the fire using only words. You could have a million people tell a child about what fire is like, the dangers of it, the power of it, the pain of it. But they will never develop an innate understanding of it that helps them navigate the real world.

Until they stick their hand in the fire. Then they know.