Yes, the concern is that in a direct democracy demagogues will be elected far more often and with far more extreme views.

Bicameral parliamentary systems are even less prone to this since the prime minister is elected by the ruling coalition. But they have other problems.