The problem is different for encryption schemes relying on hardness of integer factorization.
> So it's still hard, and will always be
That's a pretty bold claim, considering we're just beginning to understand quantum phenomena. Plus you have zero clue about what else awaits us in the future.
Solving 1024-bit key in SHA (or sth) will take as much as 512-bit key using a comparable traditional computer. So it's still hard, and will always be.