I'd argue they were about like us, no more, no less. Hunting mastodons was actually very sophisticated technology - I mean, all they had were sticks and rocks, against a beast that could easily kill them. On the other hand, disease, malnutrition, and untreated injuries really dragged them down.
It's important to divide between history and pre-history, though - living conditions for stone age tribes were very different than for civilizations, which had agriculture and permanent structures and all the benefits and problems that go with those things.
I’m pushing back against OPs idea that humans weren’t agile for most of history. Hunter gatherers were about as tough as humans get.