Once you do that you lose the stateless benefit of cryptographically signing the object, and at that point you could just store the whole object and give the user a lookup id to the object without the complexity of cryptographically signing anything.
So that could be the difference between whatever you're using for storage, and a tiny, fully replicated in-memory structure. Depending on your overall scale, of course.
Only if the collision of lookup IDs (accidental or malicious) is effectively impossible. If it's possible to generate a collision, then you've thrown away your security.
This would also effectively give every server the ability to issue auth tokens (and mutate them in the DB), which is not a great choice if you care about security. But if you're handing out unsigned lookup IDs, you probably don't.
And actually, yes, if I were using 128 byte identifiers for some reason, I would still be concerned about malicious parties being able to cause collisions. Securely generating random numbers does not seem easier than authenticating.
Generating 128 byte identifiers is ridiculously overkill. I can safely say if you generate 128 byte identifiers securely, the earth will cease to exist before a collision is found.
You would be safe generating cryptographically secure 128 bit identifiers and looking them up. It is trivial to generate IDs that will not collide.
Signing avoids the need to generate secure IDs, though, and can also avoid hitting to the DB for expired tokens. (Load from expired tokens is probably not a major concern, though.)
If you want a stateless cookie, you won't be able to revoke it down the road. Which means if the cookie is long lived, someone who gets their hands on the signed cookie can impersonate a user for a long time.
If you cryptographically sign a cookie and it isn't stateless, you haven't really improved anything because you still have to look into up.
Cryptographically signing a cookie is great, especially for distributed applications, but it does have some properties that might not be desirable.