What matters is your path through spacetime. The four velocity is just a tangent vector of this path. The magnitude of the tangent vector has no physical meaning, only its direction does.