All cryptography exploits the inherent difficulty of factoring integers. POW requires brute force precisely because the underlying hashing algorithms are built around large primes.

This is, actually, how I layperson-explain cryptography: there’s no fast or easy way to take any huge number and know what two numbers mutiplied to make it, and this mathematical property is what makes (good) passwords hard to crack.

That is simply not how cryptographic hashes work. They have nothing to do with primes, and having a quick way to factor large numbers would do nothing to impact the security of the PoW part of Bitcoin (I believe it may affect the security of your wallet, though, but that is an entirely different attack).

Note that even being able to quickly reverse the hash function for Bitcoin wouldn't do anything to the PoW security.

The only thing that matters for PoW as implemented in Bitcoin is that there is no way to predict the value of the hash of a block + nonce faster than computing the hash. This doesn't rely on integer factorization difficulty in any way, it simply relies on a construction that uses many one-way functions.

This is very not true.

Only RSA is built around factoring. I suppose if you consider discrete log to also be factoring related, some other public key & key agreement algs are also.

Hash functions and symmetric algorithms are not based around factoring.