I think the distinction between the two has to do with the primary driver of the book. Genre fiction tends to be about telling a story about people and events. Plot and character both get strong billing, but plot dominates. In literary fiction, though, plot tends to be unimportant and character analysis dominates. A whole book can take place inside the mind of one person over the course of, say, one uneventful day.

This isn't an iron-clad rule, of course. By my definition, A Wizard of Earthsea is basically literary fiction; not coincidentally, English teachers sometimes assign it. Occasionally, a writer sprinkles fantastic or pulpy elements in a work of literary fiction and earns high praise; Gabriel García Márquez and Salman Rushdie do this routinely. It was kind of amazing that Kazuo Ishiguro's Never Let Me Go — very much a dystopian science fiction novel — made the Booker shortlist in 2005.