Indeed. Not just shared, but shared across security domains, e.g. across OS processes, or across the kernel / userspace boundary.

Speculating within one security domain would not be problematic, because anything such speculation allows to glean is already accessible normally. This is sort of similar to having a single reference to a mutable object, which is also safe.