News for hackers. Stuff that matters.
Re:The ~200 Line Linux Kernel Patch That Does Wonders by adobriyan on Tuesday November 16, 2010 @10:52AM (#1910445)
make -j$(2N + 1) is roughly where minimal compile times are.
Re:The ~200 Line Linux Kernel Patch That Does Wonders by jedbrown on Tuesday November 16, 2010 @11:07AM (#1910490)
Where N is the number of physical cores? I do not use hyper-threading (it tends to be bad for the floating point and bandwidth limited operations that I do), but usually find minimal compile times at N+1 jobs (but with little penalty for several more).
Re:The ~200 Line Linux Kernel Patch That Does Wonders by stingraycharles on Tuesday November 16, 2010 @11:15AM (#1910519)
It depends on many factors what the optimal number of concurrent builds is, but the bottom line is that you want to maximize your CPU utilization and minimize context switching.

If you think that one extra concurrent job is enough to fill CPU utilization in the time that other jobs are blocking on iowait, then you are fine.

So, bottom line, factors to think about:

- your i/o throughput for writing the generated object files;

- the complexity of the code being compiled, - template-rich C++ code has a lot higher CPU usage versus i/o ratio

- the amount of cores in your system

Re:The ~200 Line Linux Kernel Patch That Does Wonders by azim on Tuesday November 16, 2010 @11:44AM (#1910635)
Out of curiosity, What types of applications are you running where HT hurts performance?
Re:The ~200 Line Linux Kernel Patch That Does Wonders by jedbrown on Tuesday November 16, 2010 @12:30PM (#1910804)
Sparse matrix kernels and finite element/volume integration. For bandwidth-limited operations, it is sometimes possible to get better performance by using less threads than physical cores because the bus is already saturated (for examples, see STREAM benchmarks). For dense kernels, I'm usually shooting for around 70 percent of peak flop/s, and any performance shortcomings are from required horizontal vector operations, data dependence, and multiply-add imbalance. These are not things that HT helps with.

Additionally, HT affects benchmark reproducibility which is already bad enough on multicore x86 with NUMA, virtual memory, and funky networks. (Compare to Blue Gene which is also multicore, but uses no TLB (virtual addresses are offset-mapped to physical addresses), has almost independent memory bandwidth per core, and a better network.)

Re:The ~200 Line Linux Kernel Patch That Does Wonders by adobriyan on Tuesday November 16, 2010 @11:46AM (#1910645)
> Where N is the number of physical cores?

Yes. Dunno about HT, never used a box with it.

Re:The ~200 Line Linux Kernel Patch That Does Wonders by maggit on Tuesday November 16, 2010 @11:17AM (#1910526)
I have 4 cores with hyperthreading enabled (so 8 "threads"), and find that -j10 is the fastest.