If you've never seen or heard of a person that shares a major trait with you doing a particular thing, but you know that people with that characteristic are common, you probably don't think that you can do that thing either.

In addition, it's not at all unreasonable to think that way. Why waste your time on something that you will ultimately likely not be allowed to do? That's good decisionmaking, not "self-imposed race-based limiting factors."