Stop estimating single points and start estimating ranges. How long will something take in the best case scenario, the typical case, and the worst case. That way your project manager can capture the risk, which is arguably a much more important factor than time. If a task has a high impact on the project and there's a big difference between the best case and worst case times (eg it's a high risk) then it needs more attention and resources.