I took GP as more about data processing than dualism. A language model can take language and process it into probable chains, but the point is more along the line of needing to also simulate the full body experience, not just some text. The difference between e.g. a text-only game, whatever Fortnite's up to, and real meatspace.

And furthermore no simulation can have a “what it’s like to be them”

No it's not, it's an assertion that there is an essential biological or chemical function that occurs in the brain that results in human mental phenomenon. It has nothing to do with a soul. That's ridiculous.

Here's a more academic argument, although not quite mine: https://www.degruyter.com/document/doi/10.1515/opphil-2022-0...

If consciousness is a computation (and I think it is), and if you fork() that computation (as the article imagines as its core thought experiment), you end up with two conscious entities. I don't see the philosophical difficulty.

If consciousness is substrate independent, it can never be embodied like we are. If evolution explores solution space regardless of what science understands, it's likely minds operate on laws of physics that aren't appreciated yet. It's possible that having experience requires being real. As in the computable numbers are a subset of the real numbers, and only real life real time implementations can experience, because the having of experience can't be simulated.