This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance, especially on systems with lots of cores. Similarly, there is no requirement to spawn separate threads for task concurrency. The non-blocking coroutine-style concurrency common in high-performance server software supports much higher concurrency and with context switching costs that are much lower.
The thread-per-core coroutine-style is easier to write and maintain too -- the logic is almost entirely single-threaded. No deadlocks, races, etc to worry about because lock graph concurrency is complicated.
It wouldn't. There will only be a context switch if the thread is woken up for some reason (eg. the operation it's waiting on compeltes). If a thread is spending a majority of its time waiting, then it's not going to be doing a lot of context switches.
> This is why thread-per-core software architectures are idiomatic for anything requiring maximum performance
That's a stretch. There are some specific real-time usecases where you want to lock a specific thread to each core, but those are extremely niche and imply total control over what is running on the system.
This architecture has been idiomatic for high-performance server software for a solid decade. It only seems niche if you don't work in an area of software that cares a lot about performance. The performance difference is not small, 10x throughput on the same hardware is not uncommon.
And yes, it implies control of the resources your process nominally owns, though by no means the entire machine necessarily. Again, idiomatic, and software like high-end database engines have been designed this way for a very long time so it isn't a new idea. Designing software that takes full control of its resources, and does its own I/O and process scheduling, isn't magic, just different.
A classic multithreaded software architecture will never be remotely performance competitive with a thread-per-core architecture. It isn't expected in theory, and is never demonstrated in practice. Modern thread-per-core architecture has its origins in supercomputing, specifically to enable better scalability as the number of cores per CPU (and server) became greater than 2.
I’ve spent the last two and a half years mainly programming in single-threaded NodeJS environments with a focus on heavy same-task concurrency, and I’ve still run into deadlocks and races, despite the lack of OS threads involved.
However, your point that context switching is expensive, and is why thread-per-core architectures are so common, still holds.
In an OS like Windows, on "typical" hardware, the amount of memory per thread for stacks and bookkeeping the OS and/or userland allocate is usually negligible. In this context also remember that the reserved stack sizes per thread do not refer to physical memory but virtual memory (or "committed" vs "reserved"). A typical thread usually commits 1 page for the stack and the equivalent of up to 2 pages worth for any bookkeeping[0], etc, which would put the memory committed for the 7412 threads the author mentions in the ballpark of maybe 30-100 MiB of memory actually committed.
The OS scheduler should be good enough to distinguish between runnable/waiting (sleeping) threads, eliminating the need to even look at most threads (which will be in waiting state) until some signal arrives putting them back into a runnable state.
You only really get a problem when you have too many runnable threads (fighting over the available resources) or when too many threads keep rapidly toggling between runnable and waitable, as either case would potentially lead to a lot of expensive context switches (and the former case is an indication you're trying to do too much with the physical resources available).
[0] Tho some things may commit considerably larger amounts of stack space and/or bookkeeping space. E.g. if I remember correctly java8 commits 1MiB of stack space per thread by default, while e.g. java11 merely reserves that 1MiB.
A single thread can then wait for events that would require some processing (IO or timers being the most common) and then it passes that unit of work to the thread pool to be processed. Windows even takes pains to ensure that the last thread to process a unit of work, might then also be used to process the next item in the queue of work to eliminate costly context switching, and invalidation of data/instruction cache.
Thread pools allow the best use of threading by: - limiting the number of threads a process creates/maintains. The thread pool itself can grow or shrink as demand changes. - limiting the number of threads that are in a runnable state by queueing work to the pool. The pool itself will be sized according to number of CPU cores, e.g. 2 * CPU cores.
The main idea behind tread pools is that any thread that's not doing useful work is a waste of OS resources, so they make sure that they are either always fed with work, or get destroyed when idle for too long. From an overall machine efficiency perspective this makes total sense.
Having hundred or even thousands of threads created and idle is massively wasteful and bad design, the OS should not be burdened with such housekeeping.
Note that Apple's GCD also works on the same principle, as does libev/libuv.
The number of threads should only become a bottleneck if you have less than 0.01% of them idle.
Agree, because spawning a new thread and killing a thread carries its own overhead so better to have the thread sleep or be paused by the OS and cpu scheduling.
Its all about getting the balance right, but I have noticed dont net apps run their gui on a single thread instead of a window(s) linked to a thread, like in a multi document interface (MDI) setup.
This is where Moore’s Law has brought us: if we want better performance, we need to take advantage of parallelism. Thus more threads.
Threads need to scale with CPU cores, however simply increasing #threads doesn't help. You need to split tasks of your program such that more tasks can be executed in parallel while not adding more overhead than you gain by parallelism.
Then you can distribute those tasks on as many threads as possible (while still being bound by #cores).
It's bound by the slowest shared resource in the system (Amdahl's law). That could be disk I/O, memory bandwidth, or power budget.
Seems like that is an ideal activity for a thread, and in fact why most people use them? Combining all those waits into a single thread/process means also creating a dispatcher which then either spins up a thread to do work (ahem), or blocks all other useful work. If the cost is low (and for thread it usually is), having lots of blocked threads often makes the code easier to write and maintain.