Caches are transistor-dense, it's hard to optimize them further. Maybe something like Transmeta's architecture, a hybrid software/hardware approach would yield some improvement.