I get it, but come on - this was not a "performance optimisation" issue, but one of bad architecture; an architecture that certainly doesn't inspire confidence in the priorities you mention: accuracy, simplicity, safety.
Now, in addition to probably optimizing what they've done, converting to, for example another container system, like K8s where they can scale vertically a bit better may have been another approach.
The biggest issue that I see is gRPC doesn't work that great with Node. You can't use it with cluster which means you have to self-manage threads/processes and it adds complexity there. Yeah, there's definitely issues that come into scaling in terms of performance optimization.... but where they started from isn't unreasonable imho.
That doesn't prevent us from thinking about performance. GTFO with this nonsense.
I would have probably pushed for a shift in orchestration to kubernetes along with some tweaking as an initial uplift. Others would re-write the whole thing in another language. They chose to add a bit of complexity for multiple requests per node support. They all have their pluses and minuses, but in the end it doesn't mean the initial approach was bad, or their refactor wasn't pragmatic or practical.
Dramatic rewrites to a codebase lead to instability and in practice fail as much as succeed.
Otherwise known as re-architecting?
The simplest solution is to scale to one worker per node initially if you're doing anything compute intensive... once you've done that, and/or you need better performance for any number of reasons including cost, then you can do more. Now, I'm not sure I would have gotten to 4k nodes before I started to re-evaluate parallelism or better scaling options, but the initial implementation is absolutely fine.