Yes, you can do the same math if your stories are consistently sized, or if you have a sufficiently large data set. All you need is a consistent unit that is related to complexity, risk, and time. The less precise that relation, the larger sample size you need.

For many teams who get requests from external stakeholders in widely varied technical environments - ie consulting, often - estimation is functionally just a conversion process to a consistent unit. But you're absolutely right that for some teams the stories themselves are a good enough unit.