You can use this argument against anything not done yet: "we've been trying to build fusion reactors for 50 years now, we should give up now"

Except that isn't the argument I am making.

We have small quantum computers, we don't have big ones.

We have fusion power on small scale, we don't have it on commercial scale.

Seems to be the same argument.

We have small, usable quantum computers and have grown them for years.

We have never had fusion power on a small scale (we have had fusion ignition/reaction, but never [afaik] generated usable power), let alone showed continuous growth in scale of that power over the years.

They're not the same. Fusion is a 'science' problem, quantum is (mostly) an 'engineering' problem.

Re-read my comment and then point out the spot where I advocated ceasing research and development of anything.

The possibility that good-enough quantum computers might not be possible, is just an additional argument for not replacing the current key exchange methods with post-quantum methods, but at most only with hybrid methods, which retain at least one method for which there exists reasonable trust.