The term is slightly controversial but usually refers to works that emphasize prose over plot and the interior feelings of the characters over setting or action. More often than not the writing is in some way intentionally difficult, the paragon here being James Joyce. Critics say this is pretentious but fans enjoy being challenged by the writing. If you read something by Harrison expecting Heinlein you'll find rich emotions, a confusing plot, but unlike mainstream literary fiction the setting will have sci-fi elements (or fantasy or both). The range of these works is wide: Harrison definitely, Ballard. Most people would include Dick. Vonnegut said he was re-categorized after writing Slaughterhouse Five even though his writing didn't change at all.