To be clear, we agree on this.
My point was that, in a direct democracy, it's not just the public's ends that are paramount, but also the public's preferences on means.
One good example of what I'm talking about is Ancient Greece, which was a direct democracy, despite having so-called representatives. In the Ancient Greek system, people were chosen for the civic duty of governing the country like they continue to be chosen today for jury duty: they were drafted by lottery. They weren't afraid of politicians betraying the public will, because any politician that did anything the public didn't like, would be killed.
And to be clear, "anything the public didn't like" included, for example, being friendly to foreign diplomats, or accepting gifts, even if these things would have had no later bearing on their political decision-making, i.e., on their practice of their profession.
In such a context, a politician is forced to not only act toward the will of the people, but also to "perform politics" in a way that avoids upsetting their citizen-base, and so to avoid using some of the most effective political tools because the public finds them uncouth. Also, such performances often require the use of ineffective or downright counterproductive political tools, simply to satisfy the public.
A clear modern example of the effect of direct democracy is when a municipal elected official, e.g. a county sherriff, feels pressure to put a suspect behind bars for a crime there's not yet sufficient evidence they committed, simply because the public demands that someone be punished for the crime.
The entire point of representative democracy, is that the public is often irrational in ways that makes their proximate demands an ineffective way of achieving their own stated will. In this example, the public nominally wants to see justice done—but in the implementation of it, directly demands injustice. A representative is supposed to listen to the will of the public, certainly, but to ignore the direct demands; or translate the direct demands into the will they stand for.
Patients know their symptoms but not the causes. Users know when there's a bug but not the correct fix. Well, voters know when there's a problem, too; but they're not any better at figuring out what to do about it than patients or users are.
People don't have enough money to get by? Fixing that is certainly "the will of the people", and a politician that hears that complaint but doesn't do anything about it is certainly not being a representative. But the decision between fixing it via welfare, jobs programs, minimum wage, UBI, etc? That's statesmanship. And letting the public decide which of those policies is better is just abdicating the responsibility to practice the job of politics; just like a doctor just giving someone pills for whatever disease they think they have would be an abdication of responsibility for the practice of the job of medicine.
To give an extreme example, a personally peace-loving representative is supposed to declare war on another country if their constituency wants war to happen. However, if their constituency wants an end to hunger and some are noting that the neighbours have more food, the representative should do whatever they can to end hunger, avoiding war if they know that is unlikely to help.