The other reason being the gradually mounting paranoia eventually culminating in what I can only describe as a paranoid anxiety attack.
The day that happened I quit.
I'm pro-legalisation but I abused it and ruined it for myself. Many others do too, all the while espousing its complete lack of drawbacks, which I find irritating.
You should re-phrase this as: "the people I know who publicly advertise the fact that they smoke pot".
In my experience, there are a lot of marijuana users who are otherwise extremely normal people (and even sometimes on the exceptional end of normal). But given the legal status of marijuana, an ambitious middle-aged professional with a young family who also smokes a couple times a month is unlikely to publicly advertise the fact.
Because you (presumably) don't smoke, those people are extremely unlikely to confide in you about this particular subject.
A good counter example is someone I know who didn't start until around age 25, only indulges occasionally (so it isn't an addiction), and has very high cognitive ability (currently making 6 figures in the financial industry).
That said, I personally don't feel comfortable indulging in something chronically that has such pronounced effect on my mental abilities. Even if it has no long-term physical consequences, the habit (and what comes with it: junk food, junk tv, lethargy, etc.) itself can't be ideal.
Perhaps it would work if I didn't have access to the internet and/or lived in a more nature-like area. Or perhaps it would work if my lifestyle was already different. I don't know. But as it is, if I smoke I become either couch-locked or jittery and paranoid.
I think individual differences matter a great deal, and you're right in pointing that out.
That said, the vast majority of regular pot smokers I know are more similar to me than they are to you, so based on that I think the negative bias is at least somewhat justified. I'd never tell someone not to smoke, I wouldn't judge chronic smokers no matter their lifestyle. But I've warned every one of my siblings and those curious about it to keep an eye out for how they respond to the stuff. It's nowhere near as potentially dangerous as alcohol, I think, but it's also not quite like picking up a coffee habit.
(I do have to confirm that I would occasionally get a burst of 'creativity' in particular when coding on purely-fun side-projects! Doing so was enjoyable and funny albeit not always very productive. I miss that sometimes.)