It adds latency. You have to probe all the earlier levels of the cache before you can access the next one, if you want the caches to help with bandwidth. The cost of probing more levels adds up.