If quantum computers become more practical in our lifetimes, we don't want our current secrets vulnerable to that analysis. Scaling a quantum computer definitely isn't as simple as the evolution from valves and transistors was to integrated circuits ... but estimates from experts on the difficulty vary from "this is very very very difficult" to "this will remain physically impossible". Either way that's still more likely than brute-forcing an encryption key with modern standards; so there's a rationale for prioritizing post-Quantum security today.

That said, you are right to be wary; if there is a side-channel or implementation vulnerability in a PQ algorithm ... then you are much worse off using it. Worst-case; imagine an RCE in the PQ implementation. So it's a good idea to be cautious and check the code rigorously.