The temptation here is to let sleeping dogs lie, but I think this just makes it worse. I imagine if you don't make a lot, it's probably not worth their time to chase you, but heck if I'd want to live with that constantly hanging over my head. It's bad enough owing money you can't pay.
ADD: For what it's worth, I have a sneaking suspicion that there are a ton of people in this situation.
In some circumstances, this can even include later years in which a return was filed, if the unreported income affected the income, deductions, credits, losses, or other tax items in that later year. This is usually not an issue for individuals, as the problem usually arises from carried-over losses or credits that are generally available only to corporate entities.