Yeah, if you a very common password like "hunter2" or "Password1", then even with a KDF that takes 100ms to generate the key, it's still very feasible to run through the 100k most common passwords and compromise it within a few hours.

If we're talking a more random but still short password (for example, just 8 random alphanumeric characters is log2(262+10)8 = ~48 bits), then the KDF becomes very attractive to help skyrocket the brute forcing cost to something more similar to trying to brute force the 256 bit key instead.