> The thing that I'm less clear on is, if and when they achieve some kind of recursive improvement exponential growth thingy, isn't that going to heavily disrupt the entire economy / stock market? In that environment, would Anthropic's stocks really be a good bet, given they have to operate within the economy?

That's a long-term problem to be dealt later, greed dictates to pursue the former as long as the music is playing and everyone is dancing. Similar to 2007-2008 when it was clear that things were breaking but the incentives were to play the game for as long as possible before it imploded.

It's clearer to me why this works for generic investors than for the AI companies themselves - unless their plan is just for key employees to cash out and live on a private moon just before Consequences arrive.

>> that first reaches AGI (or whatever we decide to call it) might pay off

It is a myth. They are chasing science fiction. They either dream of an omniscient overseer who will take control of society and nominate its shareholders-parents into the governing elite, or they envision a chatbot that will solve every problem, again turning shareholders into the power brokers. They forget that nobody will ever be allowed to "win capitalism" and take over the planet. Companies have are torn down for far far less. Every AI model is data, ones and zeros. It therefore can be copied/shared/leaked. Intellectual Property law is not a ladder for assent to lordship. It can and will be rewritten or outright ignored rather than allow capitalism to collapse.

“Yes, the planet got destroyed, but for a beautiful moment in time we created a lot of value for shareholders.”