In addition to blatant racism, I suspect much of it is due to the potential for cultural and spiritual revolution should marijuana (or psychedelics in general) become legal. We can look to the 60's for an example of what can happen when mind-altering substances become widely used recreationally. There were a lot of other factors at play then, but the growth of counterculture and subsequent pushback was the origin of many of today's political battles. I can see why conservatives are afraid of that rapid change - it will be incredibly disruptive to the established order. (I disagree with their fears).