When building Chrome from source code our build system creates about 70,000 processes. This is enough that, even without CFG issues, process-creation time is a factor in the minimum build time.

Sounds like it might work much better to only have as many processes as there are logical cores.

Most build processes are going to run at least the compiler once per code file. Plus or minus a linker and preprocessor and whatever else. Nobody said all 70,000 quoted processes were run simultaneously.

You could presumably write a compiler that didn't do the traditional start - read - process - output - quit cycle to avoid paying startup costs, but it is generally assumed that startup cost is small relative to processing, and that avoiding complexity is worth the cost; if your OS makes startup really expensive, maybe that's not a reasonable assumption.

We avoid having more processes that cores running at a time. But we run a separate compiler instance for each file we compile, 'cause why not? So, 70,000 processes over the length of the build.

Compilers generally instantiate one process per input file. (.cpp or .c files, but not .h) Because the compiler writer can generally be assured that the process is very short lived, compilers are typically written to never use free(). When the object file is created, the process exits, and the memory is freed by the OS. This is generally a significant speedup.

Unit tests. When a unit test crashes, that crash needs to be logged, and it can't interrupt the other tests. The good way to do this is by creating a new process per unit test.

Some web servers in certain configurations will start one process per connection. If you have 1000 simultaneous connections, you have 1000 processes. https://httpd.apache.org/docs/2.4/mod/mpm_winnt.html

That's 3 examples. There are others.

The article literally answers that. It's very common for unit tests and building infra, especially when ported from POSIX systems.

And as for why you might want to spawn processes in these cases - it helps isolate memory corruption, other unrecoverable process failures, or even just issues like memory fragmentation to a specific unit test / file build - so that issues building/testing X don't show up as intermitent failures that get misattributed to building/testing Y.

As a workaround for perf issues like the article, you might choose to reuse the same process more instead, but it's a tradeoff.

Sorry, I thought problem is with thousands parallel processes execution. There problem is with some remainings from previous process runs.

...or with the fact that they're constantly being created and destroyed as some tests/build tasks finish and the new ones are spawned.