No, it should not be banned. The drug itself isn't the issue. If a person took precautions to ensure they couldn't harm others while using the drug (consenting trip sitter who can physically handle them, proper set and setting) then they should be able to do that.

However they must also bear the cost of any harm done in that state. The problem isn't that someone took a drug, it's that they did X thing to hurt others. Punish X thing, not the drug use.

I acknowledge that this has serious problems because not everyone acts responsibly. But the alternative is to implement a system which removes the basic freedom to control what goes into your own body... and still have people acting irresponsibly anyway.