Are you even reading the text you cite? That compares SIMD to SIMT. SIMD, not SPMD. SIMD is a more restrictive hardware paradigm, SPMD is just a mutiple copies of a single program running on multiple processes/threads, i.e., exactly what CUDA does.

Most MPI-programs are SPMD, and really, the only conceptual difference between MPI and CUDA is that a thread's id is a point in a grid space that's relevant for scheduling and memory locality. Oh wait, that's true for MPI as well, only the scheduling and memory locality relates to CPU cores and nodes, instead of CUDAs SMPs and blocks.

The only real justification for a different programming model is that SIMT creates opportunities for implicit SIMD on a thread scheduling level if you've written your program in a way that the warps don't diverge.

> Are you even reading the text you cite?

Yes I have read it enough times that I knew exactly where it was in the book.

> That compares SIMD to SIMT.

Yes and so does the op I responded to.

> SIMD, not SPMD. SIMD is a more restrictive hardware paradigm,

Thanks but I was already aware. The op was the one that conflated SPMD and SIMD, not me.

> SPMD is just a mutiple copies of a single program running on multiple processes/threads, i.e., exactly what CUDA does.

CUDA is a C language extension that enables you to write host and device code in the same source. That's it. SIMT is the model of compute on NVIDIA SMs.

> The only real justification for a different programming model is that SIMT creates opportunities for implicit SIMD on a thread scheduling level if you've written your program in a way that the warps don't diverge.

No that's incorrect. SIMT is implemented using predicated execution and instruction replay exactly so that warps can diverge.

Again, it's amazing you guys are so sure that NVIDIA pulled a fast one on just the whole community/industry and didn't actually innovate but it turns out you're actually missing some pieces of the puzzle.

and again, did you even read my reply? You're disagreeing by stating my point back to me. The point about divergence is that IF the threads DONT diverge, you get SIMD scheduling, the GPU can schedule the equivalent of a SIMD OP across the warp.

Obviously they can diverge, because this is SPMD/SIMT

OP did not conflate them, SPMD and SIMT are the same model, they're just used in different contexts.

well i guess it's just your word against ... basically any other resource on the matter (in addition to just a common sense understanding) :shrug:

Ok dude, give me an actual argument. Tell me a substantial way that SIMT is different from SPMD

Subgroup operations[1]. In SIMT, they are fairly easily modeled[2] as communication between the different threads running on the same SIMD, and in most cases explicitly expose the predication mask. In fact, subgroupBallot(true) is a common idiom to extract that mask, and also to determine the subgroup size if run in subgroup uniform control flow. Generally they have good performance characteristics.

In SPMD, subgroups aren't easy to model, and would generally be emulated by inter-thread communication if it's important to match the semantics of a source program that includes them. Performance in that case would be terrible.

[1] In Vulkan, OpenGL, OpenCL, and WebGPU they are called "subgroups". In Nvidia including CUDA they are called "warps". In D3D and AMD they are called "waves". In Metal (often running on the same hardware) they are called "simdgroups". This fragmentation of terminology is part of the flavor of working with GPUs.

[2] "Fairly easily" by the standards of GPU infrastructure. In fact, the exact semantics have never been nailed down, and in particular "reconvergence" is poorly defined. Working through this is one of the things blocking subgroups in WebGPU (https://github.com/gpuweb/gpuweb/issues/4306). Even so, they're used commonly in practice, especially for things like fast matrix multiplication where you need to shuffle lots of data into place.

This is true to some extent, although certainly SPMD has subgroups and reduction operations on those are fairly common.

And you are essentially repeating my point from before, that the fact that an operation among the SIMT threads can be scheduled as one SIMD op is the only special behavior. My example was regular vectorized SIMD ops, but your example of reductions are essentially the same point. Shuffles and horizontal adds are present in SIMT just like they are in regular SIMD.

These same reductions exist in contexts where the model would traditionally be called SPMD. But because the concepts map onto a different level of the system we might not easily recognize them as the same. The same kind of subgrouping happens in a multinode SPMD context in terms of NUMA regions and nodes, and an MPI implementation can handle reductions on a within-node or within-NUMA-region group of ranks more efficiently than a group containing ranks assigned to multiple meaningful hardware regions.

So I agree with you that the one thing SIMT has to differentiate it from SPMD is the SIMD-like scheduling of multiple threads -- and this was exactly my point from before -- but even here we find that SPMD has the same concepts, just not as explicitly, because the stack typically used in SPMD is more flexible, and SIMT can take advantage of specialized hardware ops.

Optimizations like this do not necessarily mean that the model isn't SPMD though. It's still a single program (a kernel) processing multiple data (indexed by the thread id). The one key idea of SIMT is really all about the batch scheduling of threads, the fact that multiple threads from a warp can be executed on one SM partition (using CUDA terms here) using a single instruction and a mask.

SIMT emphasizes the GPU scheduler, but it still fits completely within SPMD. All SIMT technologies are SPMD technologies, but not all SPMD technologies are SIMT technologies. I'm not convinced there is enough new in SIMT to warrant the branding using a new architecture type in the taxonomy. They could have just as well emphasized the thread scheduling as an optimization in an SPMD context.

Please don't respond to a bad comment by breaking the site guidelines yourself. That only makes things worse.

If you'd please review https://news.ycombinator.com/newsguidelines.html and stick to the rules when posting here, we'd appreciate it. Note this one:

"Please don't comment on whether someone read an article. "Did you even read the article? It mentions that" can be shortened to "The article mentions that.""

My bad, I'll do better.

You've continued to post aggressively, attacking other users and generally posting in the flamewar style, even though we've asked you many times to stop. If this keeps up, we're going to have to ban you. I don't want to ban you, because you obviously have a lot of knowledge to share, but your aggressiveness destroys more than your knowledge contributes—a lot more, actually. Fortunately, there are plenty of other knowledgeable users here who treat their fellow users respectfully. Please be like them instead.

https://news.ycombinator.com/newsguidelines.html

It pointless trying to abide by your rules of decorum - I call a thing a rant - as if it isn't and that makes me aggressive? In reality you're just mad I pointed out your kafka-esque moderation policies.

> Please be like them instead.

Be an arrogant martinet? No thanks. But I'll stop posting so congrats on keeping your community homogeneously obnoxious (and ignorant).

I don't remember anything you pointed out about moderation, and in any case we don't ban accounts for criticizing the mods.

This isn't a response to just one comment, let alone one word, but to the pattern of your comments in general: they stand out as nasty, abusive towards other commenters, and not in the intended spirit of the site; not just by a little, but a lot. It's not a borderline call.

there's a single questionable word in the flagged comment (the word `rant`) and it's not a borderline call? yup totally makes sense.

You're continuing the pattern here.