I look forward to the day when spaceships become as mundane as trains and helicopters, so that their presence alone will not make a novel "science fiction".
Le Guin never identified her books as science fiction. Sadly the world can't seem to look at a book without shoving it into the bounds of a familiar genre. This doesn't help readers, either, as GP was apparently given the wrong expectations by these labels.
My observation is that most sci-fi is set in a world mostly similar to ours, except for one aspect/tech that's dialed to 11. Fantasy has a much looser attachment to our world, and sometimes the weirder, the better. Also, each has its own genre-tropes that can be used (or subverted) but barely translate to the other - I'm yet to see a fantasy story that is grounded in egalitarianism.
I don't remember where I read it but what you just described is one of the best strategies for writing scifi that many of the most well known authors used. I'd wager it is the defining quality of science fiction from a historical perspective: it's a form of fiction that split the difference between vanilla fiction and fantasy that just happened to grow popular in the 20th century's golden era of scientific research. This was the era of boundless optimism with flying cars and the World Fair's Futurama so a fuzzy genre that took our world and changed one abstract thing or dialed it up to 11 (alien invasion! total surveillance! AI! galaxy far far away!) became science fiction because authors chose convenient plot points based on the cultural zeitgeist of the time.
On the surface the difference between science fiction and fantasy isn't all that big ("Any sufficiently advanced technology is indistinguishable from magic") but I think fundamentally they're different approaches to imagining a literary universe. The former is based in reality but the "science fiction" part is a plot device to implement the one aspect of the universe the author wants to change, whereas in fantasy it's supposed to be a new universe whose rules/laws are only vaguely related to our own because the author just happens to also be human.
Edit five minutes later: Found it! Philip K Dick wrote in a letter (from The Collected Short Stories Volume 1):
> I will define science fiction, first, by saying what sf is not. It cannot be defined as "a story (or novel or play) set in the future," since there exists such a thing as space adventure, which is set in the future but is not sf: it is just that: adventures, fights and wars in the future in space involving super-advanced technology. Why, then, is it not science fiction? It would seem to be, and Doris Lessing (e.g.) supposes that it is. However, space adventure lacks the distinct new idea that is the essential ingredient. Also, there can be science fiction set in the present: the alternate world story or novel. So if we separate sf from the future and also from ultra-advanced technology, what then do we have that can be called sf?
> This world must differ from the given in at least one way, and this one way must be sufficient to give rise to events that could not occur in our society -- or in any known society present or past. There must be a coherent idea involved in this dislocation; that is, the dislocation must be a conceptual one, not merely a trivial or bizarre one -- this is the essence of science fiction, the conceptual dislocation within the society so that as a result a new society is generated in the author's mind,
> Fantasy involves that which general opinion regards as impossible; science fiction involves that which general opinion regards as possible under the right circumstances.
> Thus "good science fiction" is a value term, not an objective thing, and yet, I think, there really is such a thing, objectively, as good science fiction.
IMO Le Guin and someone like Heinlein were two sides of the same strand of SF. They're both moralists rather than technical speculators, and in their own way they have a didactic streak where the morality is confidently presented but rather heavy-handed.
Although The Dispossessed is politically ambiguous, it's not ambiguous in an unfathomable, mysterious, or unconfident way. Neither is something like Dune, or the Culture novels. Or even Asimov's Foundation.
Part of the appeal of these authors is that they seem quite sure of their political and anthropological sophistication. While their characters may struggle, there's never a sense the authors feel they could be completely blindsided by human or alien anthropology, or steamrollered by non-human influences which are incomprehensible and impossible to deal with.
This is what I explained to my high school teacher when she talked sf down because I wanted to write about Alastor by Vance.
I guess that could be either fantasy or science fiction, but since it has spaceships it must be the latter.