You only need a cryptographically secure RNG to generate a key pair. Assuming asymetric keys. That private key can then be used to sign many many JWTS.

Whereas generating new session ids will always need fresh entropy.

That depends on the algorithm, some depend on randomness like IVs.

Maybe you are thinking of JWE?

If the signing of the JWT wasn’t deterministic, then it would be hard to verify.

Correct. Some auth systems use JWE. Some even use it interchangeably with just JWS.