That's definitely not true for a large number of science fiction stories. The science is often a core component and can't be substituted for anything. What does Star Trek look like in a fantasy setting? It doesn't work at all.