I sometimes feel bad to have a prejudice against some pople but realistically i don't have the time and energy to sift through a lot of bad ideas when I can talk to other people who usually have good ideas.

How do you know they're bad ideas if you have not even listened to them?

this is classic exploration-exploitation tradeoff, like a multiarmed bandit. The general gist of most multi-armed bandit strategies would agree to listen more and more to people with good ideas, and spend less and less time (approaching but never reaching zero time) listening to people whose ideas you do not find rewarding.

Always be listening. But when seeking people out and prioritizing who to spend time with, consider their track record.