Ironically, the two main legal drugs (excluding caffeine) - tobacco and alcohol - have some of the worst long term effects. Others can have long term effects though, like MDMA is somewhat neurotoxic but not to a degree that matters with infrequent acute use, but it becomes very relevant if someone were to use it regularly.