With that said, if your "key" actually consists of 128 bits (16 bytes) of secure random data (with all the conditions associated with that), then you're right: the choice of KDF is mostly immaterial, so long as it preserves entropy. That's why, for example, if you're just generating a symmetric key for bulk encryption, you just grab (say) 128 bits of random data and use that as the key directly with no KDF involved.
KDFs really only enter the scene when: (1) you want to generate multiple keys from one "master key" or (2) you want to generate "secure" keying material from "insecure" keying material (say, a user's password).
The choice of KDF, however, mostly depends on the strength of the original keying material. For instance, if you have 256 bits of secure random information as your keying material, using something like scrypt is unnecessary; something like HKDF would be more appropriate there. (No one will ever brute-force your 256-bit value. It's just not happening: ever.) On the other hand, where the original keying material is weak, like in the case of passwords, you want to use something that prohibits rapid guessing, e.g. scrypt.
You are wrong about the key derivation though, and here's why: A key-derivation function that is very cheap and fast to calculate means it is easy for an attacker to brute-force lots of passwords to find one that matches your key. Using a slow, expensive KDF makes an attack much less feasible, by a factor of a thousand or more.
If you're using AES-128, if you have a random password with a 128 bits of entropy it shouldn't matter what key derivation function you use. That means, if your password can contain any printable ASCII character, a password length of 20 would be sufficient [1].
The huge caveat here is that the password has to be random, and most people are either incapable of remembering such a password or have no way of securely storing it.
But I have no qualifications in this area, so don't take my word for it...
[1] http://en.wikipedia.org/wiki/Password_strength#Entropy_as_a_...