I believe "Spectre" is an ever-growing series of ways to read the memory contents of another process by abusing a CPU's ability to do speculative execution. It's unlikely that a CPU will be available in the near-term that's immune to it, so the only sure fix is to use air-gapping. So in principle, your suggestion is vulnerable. Alternatively, the problem may be academic. There is a still-growing class of Spectre attacks, so it's hard to know how flawed your comment was.