The best Visual Designers have superb pattern-recognition skills for applying principles they've become familiar with in their education and professional work. They can quickly recognize a certain type of problem, and call on their depth of experience to apply a solution. But even the best can slip into habit of using well-worn tricks in situations where perhaps a different approach would be more fitting.
There are many sites that work well even though they look like shit. Craigslist is a good example.
You said "poorly looking sites" can't work well. But Craigslist works fine, even though it's visual design is as homely as anything you'll find.
The point is that "visual design" is not "design". If you don't like my example, surely you can think of others.
Dangerous to separate these. Poorly looking sites have poor experiences. Of course, aesthetics are subjective, so I'm speaking generally here.
My philosophy for design is to always, without exception, strive for a good looking and functioning site. You can't have a site that works well that looks like shit. Conversely you can often find sites that look great but are absolutely unusable.