Another interesting example is audio synthesis. It is interesting because there are at least two models for audio synthesis: single sample processing (as used in, for example, VCV Rack) and block structured processing (as used by every plugin API and thus every DAW-that-runs-plugins). But it gets even more interesting because the degree of potential parallelism is entirely controlled by the signal routing pathways chosen by the user. If the software is processing N independent signal pathways that do not meet until a final output stage, you have a high (well, "N") level of potential parallelism, and if N can expand to a high value then you will always be able to benefit from more cores.

However, if the signal pathway is more convoluted (no pun intended for my audio nerd crew), then not only does the degree of parallelism decrease but the issue of single-sample vs. block structured can become much more important. In the single sample case, the cost of computing a single sample has (relatively) enormous fixed overhead if the code has any serious level of modularity, but is still very cheap compared to inter-thread and inter-core synchronization. Spreading the computation across cores will frequently, but not always, slow things down quite dramatically. By contrast, block structured processing has a much lower per-sample cost, because the function call tree is invoked only once for an entire block of samples, and so the relative cost of synchronization decreases.

This sounds like a slam dunk for block structured processing.

Problem is, there are things you cannot do correctly with block structured processing, and so there is always a place for single sample processing. However, the potential for parallelization and the level at which it should take place differs significantly between the two processing models, which opens to door to some highly questionable design.

The short version of this is that audio synthesis in the abstract can always expand to use any number of cores, particularly for cpu-intensive synthesis like physical modelling, but that in the presence of single-sample processing, the cost of synchronization may completely negate the benefits of parallelism.

To add onto this very good explanation from Paul, in practice, audio processing graphs always rapidly decay into serial processing after starting out massively parallel. Most of the time, we are writing to a single output (i.e. speakers via your audio interface or a file). On top of that, some of the heaviest weight DSP happens on this final output stage (mastering chains with multiband compressors, linear phase equalizers, limiters etc.) So every 5.3ms (256 sample buffer @ 48kHz sample rate) we start massively parallel processing all the leaf nodes (audio tracks with sound files, virtual instruments and synths) and end up bottlenecking as the tree collapses into a line. Then we are stuck doing some of the most CPU intensive work on a single core since we can’t process the limiter DSP plug-in until the EQ finishes its work, for example.

We need to meet our real-time deadline or risk dropping buffers and making nasty pops and clicks. That mastering stage can pretty easily be the limiting (hah) step that causes us to miss the deadline, even if we processed hundreds of tracks in parallel moments before in less time.

The plug-in APIs (AudioUnits, VSTs, AAX) which are responsible for all the DSP and virtual instruments are also designed to process synchronously. Some plug-ins implement their own threading under the hood but this can often get in the way of the host application’s real-time processing. On top of that, because the API isn’t designed to be asynchronous, the host’s processing thread is tied up waiting for the completed result from the plug-in before it can move on.

Add on that many DSP algorithms are time-dependent. You can’t chop up the sample buffer into N different parts and process them independently. The result for sample i+1 depends on processing sample i first.

Great write up!

What I like about Pd is that you can freely reblock and resample any subpatch. Want some section with single-sample-feedback? Just put a [block~ 1]. You can also increase the blocksize. Usually, this is done for upsampling and FFT processing. Finally, reblocking can be nested, meaning that you can reblock to 1024 samples and inside have another subpatch running at 1 sample blocksize.

SuperCollider, on the other hand, has a fixed global blocksize and samplerate, which I think is one of its biggest limitations. (Needless to say, there are many things it does better than Pd!)

---

In the last few days I have been experimenting with adding multi-threading support to Pd (https://github.com/Spacechild1/pure-data/tree/multi-threadin...). With the usual blocksize of 64 sample, you can definitely observe the scheduling overhead in the CPU meter. If you have a few (heavy-weight) subpatches running in parallel, the overhead is neglible. But for [clone] with a high number of (light-weight) copies, the overhead becomes rather noticable. In my quick tests, reblocking to 256 samples already reduces the overhead significantly, at the cost of increased latency, of course.

---

Also, in my plugin host for Pd/Supercollider (https://git.iem.at/pd/vstplugin/) I have a multi-threading and bridging/sandboxing option. If the plugin itself is rather lightweight and the blocksize is small, the scheduling overhead becomes quite noticable. In Pd you can just put [vstplugin~] in a subpatch + [block~]. For the SuperCollider version I have added a "reblock" argument to process the plugin at a higher blocksize, at the cost of increased latency.

"For most scenarios, you really don't need all those damn cores. They are almost certainly getting in the way of you solving your problems." This rings true. I still think the most promising multi-threading strategy is to, by default, keep everything on one core and if you find some expensive and more or less independent workload to put that in another thread. This is the old-fashioned way though. All of the cool kids will push for thread pools where every workload will jump from core to core and back again.... This potentially can require a lot of locking, though, which also is not good for performance.

It's an interesting reflection then on the Actor paradigm where you are encouraged to create large numbers of fine grained tasks and toss them into a giant thread pool blender because light weight threads are "free" now. It would be quite interesting to see the consequences of thread affinity loss in that setting.

I believe Rust is pulling on the right thread here but I have no idea how this will look in fifteen years. At some point we will need to be able to tell that two threads don’t contend and thus can be “farther away” from each other, but these threads are constantly communicating and need to use the same complex and/or memory banks.

I think it could be something built upon lifetime analysis. I think a sufficiently complex compiler could use the data available already today to infer thread affinity and other rules.

Basically what I was thinking. In the near term it doesn't even have to be that precise to have a benefit (Rust builds on escape analysis that helped with distributed garbage collection, even when it wasn't that exhaustive). Even ranking threads by suspected degree of interaction is a lot of data.

These definitely don't talk. Offload. These definitely talk. Colocate. These probably talk. Bin pack.

Games are a prime example of systems which must take advantage of all possible hardware resources.

Games similarly struggle with multi core/ccx designed chips.

Battlefield 4 was famously terrible on Ryzen when AMD first released it.

To be fair, Ryzen mostly underperformed when it was released which was not AMDs fault. When for years engineers used Intel as their benchmarking and testing machines and were tweaking all sorts of parameters to measure against Intel-only, you inevitable end up with an optimized implementation that resembles the hardware details of the Chip in some kind. Any new architecture would perform badly, except if it is a perfect clone of the original testing hardware.

I wasn't implying it was AMDs fault, but the idea that the games were optimized for Intel doesn't hold true IMHO because they were also built for consoles that were AMD.

The majority of the performance was down to not accounting for CCX scheduling

The fact that we believe this is probably part of the problem with game development today.