Maybe this is anecdotal, but I feel this is dependent on age. What you describe would have been true when I was younger. But nowadays I know quite some people who never used cannabis / hardly drink any alcohol, and still tell me they'd like to try LSD or shrooms some day.