I have an idea that might help though: if logical qubits are so difficult to build and require thousands of times more physical qubits, and physical qubits are also difficult to build, perhaps we can simulate physical qubits with thousands of times more "ultra-physical" qubits.
Regarding claims of factoring small numbers: First note that error correction becomes exponentially more important for longer computations. Second, see this
https://arxiv.org/abs/1301.7007
Regarding building a tower of ever less noisy qubits: it turns out there is something called a threshold theorem which requires a certain minimal level of noise before concatenating error correction schemes makes things better rather than worse.
https://en.m.wikipedia.org/wiki/Quantum_threshold_theorem
Reaching this threshold for physical qubits in the laboratory is perhaps the primary goal of experimental quantum computing.
https://www.research.ibm.com/ibm-q/resources/quantum-volume....
With hundreds you can already speed-up chemistry simulations, what will have a much larger impact than breaking RSA.
You can? How?
So it's possible to simulate 25-qubit-level chemistry with a laptop, 50-qubit-level chemistry with a supercomputer, and 100-qubit-level simulation with a supercomputer is impossible and will be impossible for a very long time.
But going from a 50-qubit quantum computer to a 100-qubit one may take just 2 years (or 4, it doesn't really matter), and then another 2 years to get to 200 qubits, and so on. Quantum computers will solve chemistry problems in minutes that would otherwise have taken 100 years to simulate.
I don't think it's a given that just because I want to simulate a quantum mechanical system in some way, that a quantum computer is a useful tool. Or it might be, I don't know. I just don't think it's enough justification that there happens to be the word "quantum" in both the problem description and in the name of the computer.
See this paper for an overview https://arxiv.org/pdf/1203.1331.pdf