This doesn't sound right, but perhaps you are in an atypical part of the world.

40 years ago you would, I expect, find a larger variety of apples, most of them heirloom (ie. open pollinated, with a history you could trace back several hundred years, and were free to propagate at home) at the 'average grocery store'.

Now you'll typically find a selection of apple varieties that were bred relatively recently, selected for shipping and long storage life (ie. ability to retain colour, sweetness (not necessarily flavour), resistance to bruising, consistent colour and shape, russets are eschewed as they're not currently popular, etc).

To the east of London, UK, there's a (somewhat) government-run farm at Faversham, where they are maintaining a mighty collection of apples - somewhere around 2,500 distinct varieties. They also maintain around 700 cherry, and 500 pears, amongst other fruits ... many of which they can trace back several hundred years.

Complaining about just one apple variety seems to be totally missing the point - you can most certainly find, with some small effort, an apple that perfectly suits your idea of the perfect specimen.

As an American who's almost 40 and remembers grocery shopping with my mom as a young child, there were typically only three common apple varieties available: red delicious, golden/yellow delicious, and granny smith. That's it.

Nowadays, even at my crappy Food Lion, I can find those three, plus usually 2-3 additional. At local higher end groceries, I can find 8-10.

At my local farmers' market, I can find 20-30 (when they're in season).

My personal favorites are Honeycrisp, Jonagold, and Pink Lady, but since Jonagolds aren't common in my state and Honeycrisps are a restricted patent and very expensive, we usually buy gala, fuji, and sometimes braeburn.

40 years ago was 1974, which was already decades deep into the homogenization of of the (American) grocery shopping experience.

Forty years ago was 1974, hardly the zenith of heirloom varieties. Maybe what you write was true 80 or 100 years ago.

How is this a paradox of choice? As in how is it paradoxical that when better alternatives exist people prefer them?

It is not a paradox; it is perhaps irony if those same people would prefer non-agri-tech fruit if you ask them (which, totally gut-reaction-based, I suspect is the case).

But many traditional varieties are much nicer than the Red Delicious... I've never eaten one right off a tree, but the R.D.s that make it into stores seem to be pretty bland and "one-dimensional" (purely sweet, no noticeable sourness or other tastes), and often have a sort of nasty bitter tasting skin.

[I notice that Fuji apples are a R.D. cross from the 1930s... But Fuji are vastly tastier (and more attractive!)...]