News for hackers. Stuff that matters.
Re:A Not-Called Function Can Cause a 5X Slowdown by xtrapolate on Tuesday December 04, 2018 @02:58AM (#18597281)
> "The first fix was to avoid calling CommandLineToArgvW by manually parsing the command-line."

> "The second fix was to delay load shell32.dll."

If your build pipeline is continuously spawning processes all-over, to the point "delay loading" makes a significant difference - it's time to start re-evaluating the entire pipeline and the practices employed.

Re:A Not-Called Function Can Cause a 5X Slowdown by wtallis on Tuesday December 04, 2018 @03:24AM (#18597452)
Do you know of a build system that can handle a source tree as large as an entire web browser without spawning a lot of processes?

It's hard to tell what, if anything, you are recommending here. Pass thousands of files to a single compiler invocation? Ignore the problems and stop trying to make process creation and clean-up faster?

[dead]
Re:A Not-Called Function Can Cause a 5X Slowdown by Dylan16807 on Tuesday December 04, 2018 @04:10AM (#18597728)
> Are they employing caching of binaries/object files? Are they running a continuous build? Which parts of the build actually take the most time to go through? Can they benefit from building concurrently on multiple machines?

So you're suggesting things they already do.

And if you're using multiple machines, you probably want to max out each machine, so it matters a lot if there's code that falls over and dies when applied to a large number of cores.

Re:A Not-Called Function Can Cause a 5X Slowdown by im3w1l on Tuesday December 04, 2018 @03:30AM (#18597491)
> Pass thousands of files to a single compiler invocation?

Sure. Or pass it a file with all the filenames. Or have the compiler work as a server that takes compilation requests over a socket. It's not like passing thousands of filenames between two processes is a deep unsolved problem.

Re:A Not-Called Function Can Cause a 5X Slowdown by gpderetta on Tuesday December 04, 2018 @03:47AM (#18597586)
Or just spawn thousands of processes which has been done for the last 40 years without particular issues.
Re:A Not-Called Function Can Cause a 5X Slowdown by mannykannot on Tuesday December 04, 2018 @08:48AM (#18599356)
So, the solution to concurrency problems is to serialize everything?
Re:A Not-Called Function Can Cause a 5X Slowdown by fit2rule on Tuesday December 04, 2018 @10:17AM (#18600106)
"Concurrency is everything serialised, properly."
Re:A Not-Called Function Can Cause a 5X Slowdown by mannykannot on Tuesday December 04, 2018 @11:53AM (#18601139)
The posts we are replying to here seem to have a very narrow concept of what 'properly' entails in this case.
Re:A Not-Called Function Can Cause a 5X Slowdown by fuzzy2 on Tuesday December 04, 2018 @08:09AM (#18599075)
The second paragraph mentions that this is about a test suite. It has to “spawn processes all-over” to do its job.
Re:A Not-Called Function Can Cause a 5X Slowdown by nwellnhof on Tuesday December 04, 2018 @09:10AM (#18599507)
Technically, it doesn't has to. You can put your whole test suite into a single executable, making it run extremely fast for C/C++ projects where process startup is often much slower than running a single test unit. This approach is used in some OSS projects I've worked on but it also has its downsides.
Re:A Not-Called Function Can Cause a 5X Slowdown by fuzzy2 on Wednesday December 05, 2018 @03:37AM (#18606594)
Some tests (e.g. unit tests) can run like this, yes. Other tests, including some benchmarks, are not meaningful when wrapped in a single process. Invocation speed and specifics matter.
Re:A Not-Called Function Can Cause a 5X Slowdown by kornish on Tuesday December 04, 2018 @03:25AM (#18597456)
Did you read the entire article? The whole point was that "delay loading" a particular DLL prevents a static analysis in the compiler from inserting hooks to perform expensive operations.
Re:A Not-Called Function Can Cause a 5X Slowdown by xtrapolate on Tuesday December 04, 2018 @03:52AM (#18597608)
> "Did you read the entire article? The whole point was that "delay loading" a particular DLL prevents a static analysis in the compiler from inserting hooks to perform expensive operations."

I actually have read the entire article. Have you?

Your explanation has absolutely nothing to do with the performance gains observed. Moreover, in the context of delay-loaded DLLs, your explanation actually makes no sense whatsoever.

Delay loaded DLLs, a linker/loader optimization Microsoft has offered since the days of C++ 6.0 (1998), simply means most process invocations in OP's case won't actually end up loading said DLLs, reducing the amount of time spent in DLL_PROCESS_ATTACH/DLL_PROCESS_DETACH (and specifically during destruction, in the Kernel).

Re:A Not-Called Function Can Cause a 5X Slowdown by masklinn on Tuesday December 04, 2018 @03:59AM (#18597662)
> I actually have read the entire article.

You provide no evidence of it, and ample evidence to the contrary.

> Delay loaded DLLs, a linker/loader optimization Microsoft has offered since the days of C++ 6.0 (1998), simply means most process invocations in OP's case won't actually end up loading said DLLs, reducing the amount of time spent in DLL_PROCESS_ATTACH/DLL_PROCESS_DETACH.

It also avoids loading gdi.dll, which avoids creating a bunch gdi objects, which avoids taking the "destroy gdi object" codepath on process termination… which is the bit that is both slow and globally serialised.

TFA's final section even demonstrates the difference it makes: 30% increase in start time including 300% increase in lock contention time but 200% increase in shutdown time including 400% increase in lock contention. The process shutdown is almost entirely serialised due to (as TFA and its predecessor explain) a system-wide lock held during GDI cleanup.