> for this to be true, the password to key transformation have to be extremely broken for you to be able to infer what is a valid key from what looks like fixed size random noise.

You're not bruteforcing the derived key (ie. the value from the KDF), you're running a wordlist against the KDF and seeing which values work.

>If your hdd is out of your device and i am brute forcing the key, who cares about the password to key transformation? that is already behind me.

The difference is that with a TPM, you can't run a wordlist attack, since password attempts have to go through the TPM, and it throttles your guessing attempts. Without a TPM you can run the KDF as fast as you want, across as many machines as you want.