You're right, eventually you have to convert points to some notion of time, if only so that you know how much work you can do in a sprint. The idea is that you make that conversion based on past observation, using a velocity estimate that represents the past performance of the team and takes into account who's on vacation this week.

There also should be an open recognition that your actual velocity will vary somewhat between sprints. One goal is to reduce that variability by improving your estimation (with practice!), but you anticipate and prepare for the variability. That's why you have a backlog (in case you need more work) and why you develop units of work that are independent (in case you need to push one).

For all these reasons it's a mistake to think of an equivalence at the level of tasks such as one day is three points or something like that.