Sort of like trying to measure a building to the nearest inch is harder than nearest foot, and every added digit of accuracy is even harder.
It's not just the number of bits that affects signal to noise ratio, though... It's also the complexity of the calculation; more gate operations and longer storage in memory leads to more quantum decoherence. And AFAIK, each type of calculation must be implemented in hardware, because reversible quantum gates must be used. This means you would need a specialized chip for each algorithm. I haven't confirmed this understanding with someone that specializes in QC, but I don't see a way around it, other than to combine many algos onto a single chip, sharing as many gates as possible, but degrading S/N further.
Something tells me, for an intelligence agency, building a quantum computer only for the purpose of breaking cryptographic keys would be a worthwhile investment.
An investment they have already made:
The effort to build “a cryptologically useful quantum computer” -- a machine exponentially faster than classical computers-- is part of a $79.7 million research program called “Penetrating Hard Targets.”
article
https://www.washingtonpost.com/world/national-security/nsa-s...
leaked document
http://apps.washingtonpost.com/g/page/world/a-description-of...
Scott Aaronson's blog has some good background information for the layman.
Doubled key size does not mean twice as hard to break. Algorithm weaknesses are a thing.
This WP is pretty accurate though. It describes the world we think will exist after legitimate quantum computing.