Otherwise, it would be impossible to have 20-point stories done in a 10-business-day sprint! Under the usual assumption that a single person is responsible for the whole story.

For the teams I've been on, a point has usually been more like a third of a day or half a day, i.e. 2-3 hours of uninterrupted concentration, and the 1/2 point card is used rarely. Sounds like you've probably used 1/2 point stories a lot more...

But this is why points are arbitrary. Each team decides whatever precise scale it wants. And it really depends on the type of work you're doing too -- whether the smallest stories tend to be things that are day-sized or things that are 2-hour sized.