Also worth noting that since that whole foods opened, the grocery situation throughout a lot of manhattan improved in that they opened quite a few more locations that also were rather large compared to the older grocery stores around.
This all said, I think the real market in a place like manhattan is for grocery delivery. Fresh Direct when they opened was tremendously popular right away, and given amazon's hold in the online purchasing space, within manhattan (and other cities/neighborhoods where few people drive), online grocery purchasing experience matters more for the brand than in store experience.
Where I think they do get it right in that sort of space is the Subscribe and Save programme which must cannibalise a fair amount of supermarket purchases. The "set and forget" nature of it works really well for stuff like nappies/diapers (it's a bit more painful for things like toothpaste because the quantities/frequencies aren't quite right).
Historically the quality of their fresh produce was also relatively poor (not particularly fresh and lower grade I think, although that just put them on a par with eg Asda or Morrisons) but I don't know if that's changed since the Whole Foods purchase (I'd like to think so).
Everything that's boxed, bottled, or bagged, I can get from Amazon. Instead of making a list I just scan the barcode and order and throw the empty one away (or put the almst empty one in the back in the fridge)
People visit mall stores sporadically enough that they might forget about the bad experience they had or hope it might be different or feel they don't have a choice.
Thus mall stores can damage their brand image for a quick buck; in fact they are often brain washed by their own advertising and have no idea how the customer sees them.
Grocery, on the other hand, has a customer that does back at least once a week, if the brand image is not upheld they can get similar goods elsewhere.
That awful whole foods at union square with the cattle chutes at the checkout might look good compared to the local bodega in NYC, but does not look good compared to a good suburban supermarket (Wegmans). It is one case where urban folk could be out of touch with the broader market.