Their speed isn't particularly relevant, what matters more is their energy. As big as that number is to us, it's still pretty insignificant; there's a lot of objects already traveling around with us in our own solar system with 70kT (per PaulHoule in other comments) of energy. A looooooot of objects. In the grand scheme of things, this is a rounding error on a rounding error. We are small.