If you use virtual addresses to index your caches, you have to clear them all on every process context switch.

With physical addresses, you just have to clear your TLB cache.