The point is: Many of these algorithms are rather new. The fact that multiple post quantum algorithms have been broken that were seen as promising shows that there is a risk with these new algorithms.
However, it should be said that the broken algorithms were in, let's say, more experimental subfields of post quantum cryptography.
If it turns out that both the classical and quantum hard algorithms are weak we're just screwed, yes. That said, at this point it's not even clear, as far as I know, that many classical algorithms are event going to be broken under QC.
https://eprint.iacr.org/2017/811
> Reassessing Grover's Algorithm
The proposal of this paper is that Grover's does not mean we need to double the size of symmetric keys to account for QM but that just a few extra bits are enough to prevent it from being efficient.
Of course, there may be new QM algorithms that don't suffer and, again, it's best to start preparing now and not later. But I think it's noteworthy for the discussion about quantum preparedness.
However, the main concern with quantum computers vs cryptography is Shor's algorithm which works on RSA and ECC. Shor's has the potential to take the problem from exponential (or at least close to exponential but sub, in the case of RSA) to polynomial O(n^3) which is much more powerful (essentially taking a log of the running time!). That is still assuming, of course, that workable quantum computers of required since will appear and that all the associated problems with that (noisy qbits etc.) are solved. Other challenges could also show up (difficulties scaling for longer computations, more passes through quantum gates, larger super-positions or even that the quantum laws governing e.g. probability amplitudes don't hold with full accuracy at such scales).
The best thing to do is to layer them, as is standard.
Which is why having as many eyes as possible is important.
Now it's very well possible that it comes back with mitigations.
And even the still OK candidates are using a young area of math.
In that scheme, you have to compromise both the PQC and ECC key agreements, independently.