On the other hand, the lectures were quite popular among grad students and faculty members, if that tells you anything.
I do think there's a bit of a misconception in that people think the Feynman lectures will teach them all there is to know about physics, at least in summary. That's false, if you want comprehensive coverage (of classical physics, mostly), try Landau-Lifshitz. The Feynman lectures are an idiosyncratic, fairly shallow treatment of selected parts of physics, produced by an extremely original mind. They're invaluable for the peculiar perspective they offer more so than anything else.
I'd recommend giving volume 1 a shot to anyone comfortable with freshman calculus and willing to work at it. Like SICP, it might change the way you think. Much as I love SICP, the Feynman lectures are greater.