Phoronix has comprehensive benchmarks on the impact of spectre mitigations if you want to find out how much of a difference it will actually make before exposing your system:

https://www.phoronix.com/scan.php?page=search&q=Spectre