In my experience, microservices/backends are almost always I/O bound, and the CPU usage is almost pure overhead (meaning it may exist, but you could reduce it with effort - it’s not intrinsic to the problem), unless the service is doing some particularly domain-specific compute-intensive task.

For your typical website backend between frontend and db, you are doing some async conversion of JSON to db call back to JSON. For an HTTP microservice you are also typically converting some JSON request body to a JSON response body with some kind of I/O call in between.

So that’s a roundabout way of saying I think that the case where majority of CPU is spent on SerDe is more common than you think. And it’s not necessarily a problem if the effort to improve is not worth the savings.

But if it's not JSON it's still going to be some SerDe. Is there another format that's enough of a factor faster that it means we should rewrite all of our stacks? In other words, the developer's time vs throwing more hardware at it.

Yes, any binary format is going to be dramatically faster than JSON. IIRC JSON was the bottleneck in k8s at one point so it was replaced with protobufs.

I think you mean system call bound which is really another way of saying CPU bound. The hardware underneath is incredibly capable, but the interfaces to it are inefficient. I’m skeptical that JSON payloads over HTTP based applications are saturating 10gig+ network interfaces.