That doesn’t mean, though, that Bitcoin would lose its network robustness; before BTC had any value at all, people were mining it “altruistically”, purely for the sake of adding auditing strength to the network so that they could use BTC to make transactions. The Bitcoin mainnet has far more of those plain-old CPU miner nodes than any other network, and they’d still be there even if the profit-mining operations cashed out.
On the other hand, if the ASIC and GPU mining pools “went dark”, it’d always be a worry that someone would pay to start them up again just for long enough to execute a Sybil attack. But Bitcoin has been “steered” away from such attacks before with community forks, and I doubt such a future would be any different.
More generally, a severe hashpower reduction could be problematic. An emergency hard fork to a better difficulty readjustment algorithm (like what other coins use) would solve that. Ideally that change would be made before it ended up being needed.
It wouldn't need to be all of the ASIC mining, of course, just a large enough portion of it to result in massively increased block times.
It's possible that point would be far beyond any conceivable price crash, though, I don't know how to put it into anything resembling concrete numbers.
Being circular doesn't make it wrong, but it does seem like a fragile basis for value.