I think i really depends on circumstances. I work from home as a freelancer which means that rather than interacting with a lot of coworkers, I mostly read things professionally from people at the top of their game. This skews the perception I have in term of skills and means that I sometimes feel like an imposter.
I didn't have that problem when I was working in a company.
Also, I don't pretend we don't suffer the same symptoms, just that without a name to put on it, it's more bearable.
Personally, I have a very hard time with compliments (I had to learn to say thank you to compliments, which make me think "bullshit" naturally). So I tend to believe I'm average, even when someone insists I'm not. That's not unlike imposter syndrome. Plus I do understand what you mean about reading stuff from/about brilliant people.
But an imposter or a fraud? Nope... I just think I don't have enough self-comfidence or self-esteem, because "being a fraud" and "self imposter syndrome" meant nothing to me not long ago, plus these terms seem absurdly guilt-inducing to me. Living without this guilt definitely helps make progress in anything.
The way I manage to get over it was to realized everybody was like me, and If I accepted others could screw up on a catastrophic scale, then I could indulge me doing it.
From this outsider position I have, it sounds like a very unlucky choice of words (imposter, fraud) for a human feeling that comes from lack of self-esteem... Psychologists should know better.
Conclusion: you can live a happy life without the "fear" of being an imposter or a fraud. Anyway, you should know when you're doing right or wrong, and that's what counts.