I have an idea that might help though: if logical qubits are so difficult to build and require thousands of times more physical qubits, and physical qubits are also difficult to build, perhaps we can simulate physical qubits with thousands of times more "ultra-physical" qubits.
Regarding claims of factoring small numbers: First note that error correction becomes exponentially more important for longer computations. Second, see this
https://arxiv.org/abs/1301.7007
Regarding building a tower of ever less noisy qubits: it turns out there is something called a threshold theorem which requires a certain minimal level of noise before concatenating error correction schemes makes things better rather than worse.
https://en.m.wikipedia.org/wiki/Quantum_threshold_theorem
Reaching this threshold for physical qubits in the laboratory is perhaps the primary goal of experimental quantum computing.