To fully close the hole you would also need to load back in whatever cache lines were evicted by those speculatively loaded. Alternatively you could provide some buffer space for spec cache lines.

No matter what you do this will result in a slowdown, either from re-loading data or from "wasting" cache space that could otherwise have been used generally for buffers.