In a healthy environment, I think "story points" (horrendous name) should approximate "engineering complexity given the current state, skill set, and knowledge of the development team." Time estimates should then be derived from that number based on historical performance of the team in its current state (if you fuck with the team structure, all bets are off and historical data becomes less useful).

Ideally a team should use points to do thoughtful analysis of their backlog and think of ways to manipulate scope, invest in cleaning up technical debt, and learn new skills to cheapen the engineering complexity and in turn deliver faster. In practice, what you described is almost always what happens. Stakeholders keep pressing for time estimates, so shit just keeps getting added to the backlog and engineers estimate "how long?" because they are neither encouraged nor empowered to think of it any other way.