An asteroid in an elliptical orbit, for example, could be a civilization killer and might currently be undetectably far away right now.
We are missing a lot right now, and we need to spend money on much better surveys.
An object on such a highly eccentric orbit would be classified as a long-period comet, which I mentioned. Yes, you can argue semantics about whether we might still call such an object an "asteroid" if it had a certain composition or primordial origin (which would be extremely unlikely for such an orbit). But there is no sharp division between asteroid and comment composition (it's a bimodal but smooth continuum) and this isn't relevant for the purposes of tracking such objects.
But that's not what I'm disputing. Asteroids routinely have their orbits mucked with by the planets and get thrown into (1) elliptical orbits, (2) non-planar orbits, and (3) ejected. In the first two cases, we aren't even looking for them because there is so much space that it's not worth trying.
Additionally, it is not true that we have found all dangerous NEO asteroids that are in ordinary orbits. NASA is still running a program whose goal is to find 90% of these [1]. For example, in just the past week they discovered a near-Earth object in excess of 1 km [2].
Of course, that's the problem with not knowing what we don't know. If there are any undiscovered, then we really don't know what percentage we know about. The fields of statistics and probabilities can probably prove me wrong, but I'm used to that.
Note that this is if they were to hit Earth. However, the vast majority of objects that size cannot intercept Earth from their current orbit. Please provide data if you have it, but my understanding is that it's considered extremely unlikely for there to be an Earth-crossing asteroid of sufficient size (> 1 km or so).
Given the nature of orbital mechanics the speeds involved are usually staggeringly large. 20km/s sounds like a lot on a terrestrial basis, but in terms of inter-planetary motion that's downright pedestrian. The Earth itself travels 30km/s around the sun, a relatively leisurely pace as it takes a full year to complete but one orbit.
Something the size of a bus travelling at a high enough rate of speed could do tremendous damage. The Tunguska Event (https://en.wikipedia.org/wiki/Tunguska_event) is presumed to have involved an object between 60-190m in size, and that's a fairly tiny object to track at any distance, akin to trying to spot a football field.
On the other hand, it's still possible that a long-period comet or rogue planet could come out of nowhere and wipe humanity out. Presumably the odds are extremely remote, judging by the rarity of extinction-level events in the geological record.