Further how do we count people who are underemployed? For instance, software developers who take fry-cook jobs because a crappy paycheck beats no paycheck. Self-employed is tricky too, because is slow business unemployed? Sure each of these edge cases is just that, but there are enough to add up to a noticable amount.
The issue is who is and who isn't included in the definition of unemployed. The truth is, if you don't have a job and you need one, but you've given up your search, you're still unemployed.