You can get really nice sound that way, but to get the full effect you need to take into account the shape of the head and ears of the listener. After all, these are important factors that help us place sound in space.

https://en.wikipedia.org/wiki/Head-related_transfer_function