Two reasons: 1) a virtual address might refer to different physical addresses (see pwildani's comment), and 2) a physical address can be mapped to different virtual addresses – a virtually addresses cache has to keep track of that somehow, otherwise the cache will become incoherent.