When you point a "0.5" and your team points a "5", stop and discuss the discrepancy until an alignment can me found. Keep doing that, every task, until you a collective meaning of "a point" is found.
Personally:
* Less than 1 - These are a smell and should not exist. These misalign incentives and encourage "bike shedding".
* 1 point tasks - no task that requires 2 people (dev and qa) is a 1-pointer, so this is something that needn't be QA'd. 1 point tasks should be rare items that can't be a subtask of a "real" task.
* 3 point tasks - These can be done in a day (depending on the team's opinion, this might be multiple people and still be 3 points). Virtually every task should be a 3.
* 5 point tasks - These are tasks that cannot be broken down to be done in a reasonable amount of time. These are generally things that have minor external dependencies. ("not hard but I'll have to get a server spun up by the unix admins.")
* 8 point tasks - These are rough tasks that haven't been fully dissected by the team and are the scrum version of a "code smell". The team should break these down and dissect them.
* 13 point tasks - "There be dragons here" tasks. Large unknowns and these tasks should not be added to a sprint until they're broken down further.
It's a one-time conversation (per new team configuration) that makes every conversation thereafter fast.
My team's pointing sessions are lightning.
Even when you do have a new team member that wants to discuss a "0.5-vs-5" point disparity, you explain the above in less time it took to read it and get back to work.
Isn't 5 points less than 2 days then?
For the record, in my team we don't use story points or measure velocity. I just had a frustrating discussion with a peer team that didn't plan an important task because "its 5 points and does not fit into a sprint".
At the round-table people would report their retired points, and I'd add them to a running tally I kept on a piece of paper next to me. "2+1+0+5+2+2+8+0+1." Then I'd record the velocity for the week for the whole team, and we'd move on.
I was already adding time to my work like an extra point or two here and there
But then in the grooming session the whole team would guesstimate even more story points! I was dumbfounded! This was beyond all of my levels of tolerance and rationality but it kept happening every sprint until something that was a 0.5 for me was a 5 point and I would have two 5 point tickets for the whole sprint that I would submit to QA one day before the sprint ended, 9-10 business days later
and actually doing that made me a star performer, what the hell!?