Except in those cases it's even more important that the NPCs don't do anything unexpected. Those NPCs are like actors in a stage play, you don't want them to come up with their own lines and confusing the audience.

Don't forget there is a certain randomness with 'more natural' and with randomness you're going to invite Murphy to the party.

Not all NPCs have to be part of a script. They can just be additional characters that add life and realism to the simulated world.

A weapons maker with a unique backstory and realistic conversations that reference it is more interesting than a bot, and opens up the possibility of unscripted side-quests.

In many cases maybe. Personally I would love to play a game with a world inhabited by "individual" NPC AI:s, where they can influence the world as much as I can, with no specific act structure or story arc.

Some significant part of gaming is risk-free experimentation in a simulated world. The experiments possible are bounded by the simulation quality of the world. More realistic NPC behavior would open up for a lot more games.

There is an older game called STALKER which had (limited) elements of what you describe: autonomous NPCs which influence the game world. Even though it was limited, the NPCs just battled for control of certain territories, I always thought it was a really neat mechanic. It made the world feel more 'real' and alive.

You would see these factions fighting and gaining/losing territory throughout the game. You could chose to help them or just pass on by, but the actions progressed regardless of your choice.

It would be fun if they ad lib