If you ask them off the cuff questions they often are wrong. But if they investigate and think about it, they'll correct themselves.
Experts are also just as susceptible to bias, motivated reasoning, groupthink, etc. as anyone else.
This is why academic papers are supposed to be a back and forth discussion. "TRUST THE EXPERTS" has a plural for reason, and should be understood to mean trust the experts after they've done a thoughtful analysis with the best available evidence and have considered dissenting expert views.
Lay people should be concerned with their opinion, but only after other experts have looked at the results/work.
The quote "I'd much rather see that transistor budget used on other things that are much more relevant," in particular, is the key to the argument. Because that seems to be basically exactly what the Apple M1 does, and look what people are saying about it.
M1 dedicates more chip-area to integer units, and also uses less power (thanks to the 5nm advantage). So what's going on with M1 isn't really a power-issue at all.
When Linus says things like:
> I want my power limits to be reached with regular integer code, not with some AVX512 power virus that takes away top frequency (because people ended up using it for memcpy!) and takes away cores (because those useless garbage units take up space).
Its clear that he was unable to be forward thinking. He was unable to see that a future chip, like Icelake or Rocket Lake, could fix the AVX512 power-throttling so dramatically. (without even going to an advanced node !!!).
Linus covered a lot of ground in that message. He was correct on his history, whether that is history you'd like to remember or not. His transistor budget argument is very plausible, but without playing out an alternative history we can't know, and by we I mean you. At the time he wrote it he was correct on the power 'virus,' and I can't think of any good reason why he or anyone else owes Intel the benefit of the doubt that some future device would improve anything.
The AVX512 fanboi contingent will forever seethe about what Linus wrote. Meanwhile Apple and AMD are eating Intel's lunch with no 512 bit SIMD instruction units at all. Linus had it right and and he is looking more correct every day.
Just so you know... I'm an AMD Threadripper fanboy, if anything. Since that's my personal computer. More importantly though, I'm a technologist. I experiment and play with many technologies, including SIMD compute, when I can.
Either way, I don't think flamebait comments like that are very helpful for discussion. But I'll try to focus on the crux of your argument instead...
> Meanwhile Apple and AMD are eating Intel's lunch with no 512 bit SIMD instruction units at all
Apple has 4x 128-bit Multiply-and-accumulate per-clock-tick on its M1. AMD has 2+2 multiply and add x 256-bit SIMD.
Those processors are actually very beefy in terms of SIMD-implementation.
There's also the Fujitsu A64Fx, which is an ARM-implementation of 512-bit SVE. In fact, 512-bit ARM-SVE is about to be widely-deployed on Neoverse.
SIMD-units are universally expanding across the industry. One of AMD Threadripper's biggest leaps forward from Zen1 -> Zen2 was the upgrade from 2+2 128-bit SIMD to 2+2 256-SIMD (doubling its throughput).
Intel is trying to stay ahead of the curve with 512, and they're really not that far ahead. Indeed: ARM's SVE is already caught up.
-------
NVidia's 32-wide SIMT is basically a 1024-bit implementation, while AMD's 64-wide SIMD is 2048-bit SIMD. With all of the "Deep Learning" going on, its actually a very important workload for a number of companies.
Linus is pretty much saying that Deep Learning / Tensorflow / etc. etc. isn't important. Which is... backwards thinking methinks. I think that deep learning is overhyped, but still I see the benefits of general SIMD compute in every-day coding situations.
For example, I think very few people would agree with: “And then suddenly he [Linus] talks about AVX512 and I wonder if he knows anything about low-level programming or optimization.”
And “But Linus is talking about power-budget”. No, you are hyper-focusing on that strawman, but his argument stands without that point. He is making a broader point based on his experience of Intel extensions over many years and he specifically alludes to that. He is saying he believes that AVX512 doesn’t add enough benefit for the costs.
You say “Linus is pretty much saying that Deep Learning / Tensorflow / etc. etc. isn't important.” but he says “I'd much rather see that transistor budget used on other things that are much more relevant. Even if it's still FP math (in the GPU, rather than AVX512)”.
You are the leaf on every comment on this thread, which is usually a bad sign of arguing a point regardless of the validity of what others say. As I read it, your arguments seem to argue about anything except for the core thesis of what Linus wrote, or you misrepresent his argument IMHO.
Edit: also, Linus is stating his opinion, an opinion which written loosely enough that you as an expert clearly can’t actually pin down as “wrong”. To make your point about the article, you would perhaps be better looking for a different example.
> No, you are hyper-focusing on that strawman
Others I've talked to have hyper-focused on that strawman. It isn't Linus I'm necessarily mad at, its his legion of fanboys who make crappy arguments based off of Linus's words.
Its not just the AVX512 example: people copy/paste his opinion on C++, Geekbench, and a whole slew of other issues constantly. Linus's arguments become memes throughout Reddit and are echo'd without analysis.
--------
But that's what happens when you put an "Expert" on a pedestal. People take their words, meme them out into new arguments, and then they don't necessarily apply anymore in the new situations.
That's my experience with "experts". The expert themselves are real, but the words they say can become warped through discussion to the point where they're meaningless.
Then again, maybe the fact that the memory and GPU are all integrated into M1 also helps propel it over the top.
How well would an AMD / Intel SoC perform if you threw a 5950X / 10900K, 16 - 128GB of LPDDR4X-4133, and a GTX 3090 into a blender and baked up an enormous die with a huge watercooled solution?
I dunno. I want to believe its the architecture that's succeeding, not the SoC by virtue of being an SoC, but I don't have the knowledge or ability to judge that.
AVX is neither CISC nor RISC. Its SIMD.
We're talking about 1970s Cray-systems or 1980s Connection Machine (CM-2 or CM-5). Its a completely different paradigm of compute. An entire history of computing that runs parallel to the CISC vs RISC wars.
> Then again, maybe the fact that the memory and GPU are all integrated into M1 also helps propel it over the top.
M1 has an awesomely wide 128-bit x4 pipeline SIMD execution backend. Apple is making sure its SIMD unit (for NEON ARM instructions) doesn't fall dramatically behind.
M1 can perform 4x 128-bit Multiply-and-Accumulate instructions per clock tick. Its no slouch.
AVX512 is better than ARM Neon on M1, but even Apple sees the importance of SIMD compute.
The ARM world is talking about 512-bit SVE instructions. SIMD compute is obscure and niche: but its a parallel niche computation style with decades of history. Its the proper solution to a huge host of computational problems.
SIMD today largely exists in the realm of GPUs: the NVidia 3090 is a dedicated SIMD-architecture for example. But as an "external" device on a PCIe 4.0 port, your CPU can only stream 30GBps worth of data to it (the max-bandwidth of PCIe 4.0 x16)
The AVX512-units inside of an Intel chip is the same GPU-SIMD architecture, but smaller. More importantly, it shares L1 cache, and therefore has TBps worth of data-sharing.
As such, AVX512 becomes useful if you need a fast CPU to calculate some things, and then switch off to SIMD-paradigm for some reason.
---------
What's a good example of this "switchoff" ?? Well, Stockfish-nnue (now default in Stockfish 12) is a great example.
You might know that SIMD-compute, in particular GPUs, are the one of the best architectures for calculating neural nets. But GPUs are terrible with branchy-code (which happens often in chess: "if(bishop) then (calculate bishop moves)" is rather difficult to do efficiently on a GPU! Its hard to explain why, but its called "branch divergence". Look it up if you feel like it).
Anyway, a CPU is best at the "if(bishop)" kind of computation. But GPUs are best at neural nets.
Stockfish-nnue beats Leela-Zero, by using AVX512 on a small-and-lightweight neuralnet, but still uses the CPU for the fast if() statements.
This is only made possible because AVX512 and the rest of the CPU shares the same L1 / L2 cache, meaning you can get TBps worth of data-sharing, instead of GBs if you went "off-chip" to another computer (like calling a GPU-to calculate a neural net).
Furthermore, the NNUE neural net in Stockfish is composed of if-statements: only PART of the neural net updates, which means it can never be efficiently implemented on a GPU. Only something with fast if() statements can implement NNUE. (EDIT: Well... maybe not "never", but it'd be non-obvious to code up a solution. I think its possible, now that I think of it... but its not done in typical GPU code these days)
GPUs / classical SIMD compute want all of its data to update uniformly, without if() statements mucking up the logic. CPUs have giant branch-prediction and hyperthreading and other such features to speed up if-statements.
-----
A small SIMD for small SIMD-calculations, that combines with fast CPU-style branch-heavy code. A nightmare for GPUs to implement. Its something only possible with CPU + SIMD instruction assist.
------
GPUs are likely still best for large-scale SIMD. Like giant matrix multiplications (or many matrix-multiplications, like in video games). Where if-statements are less common.
NV has put a great deal of recent work into getting their system to execute regular multi-threaded code with the same sort of speedup as they have had with SIMD workloads. It might be better to think of today's NV GPU as a super-threadripper when the data to be processed is already in GPU memory. The coder's job becomes making sure enough threads are ready to run to use up all the compute units at all times.
I think you're talking about "Independent Thread Scheduling", which has been deployed since Volta.
https://docs.nvidia.com/cuda/volta-tuning-guide/index.html#s...
https://images.nvidia.com/content/volta-architecture/pdf/vol...
You're close... but wrong in a few key areas.
What NVidia has done is make it POSSIBLE to run multithreaded code without deadlocks. There's still the thread-divergence performance problem.
That is to say: you can now do more classical lock/unlock paradigms on NVidia hardware, but the code is serialized and becomes sequential. One 32-wide SIMD unit will execute only 1x thread at a time in that case. (That is: 1/32 utilization. The SM is designed to handle 32 SIMD threads but is only executing one of them at a time)
So you still have the thread-divergence performance problem. On older GPUs, such divergence would have been a DEADLOCK instead. So you've gone from 0% performance utilization to 3% utilization, which means your code actually will finish.
But it doesn't mean that your code actually has any degree of worthwhile performance. Indeed: CPUs have proven that time-and-time again, per-thread branch-prediction and prefetching is best for that kind of code.
If you have a great deal of divergent if/else statements in your code, you'll want to run on CPUs. NVidia GPUs can now handle those situations better (ie: without deadlocking if those if/else statements contain thread-synchronization statements), but they aren't actually high-performance in those use cases.
I made the mistake of believing what the project manager said about it.
The question of the article isn't really a boolean, it clearly shows some experts are real and some aren't and asks the more important question -- how do we discern when the institutionalized "expert" in an area is really an expert?
Yes.
But I still think that Linus is a good character to discuss within the scope of the article. Linus is clearly considered an expert, and many people follow his bombastic words (retweeting, or using them to form the basis of their argument around here).
"Because Linus said so" is a common way to end various arguments. Linus's opinion is truly "expert", people copy it without further analysis because... well... Linus kinda deserves it. He is an expert after all.
It doesn't mean Linus is always right.
--------
The "feedback loop" of the open source world is pretty quick. If you're a bad programmer, people figure it out when they start working with your code.
But the "feedback loop" of "chip ISA opinions" is slow. There's not really any way to prove, or disprove, whether or not AVX512 is a good or bad idea.
Linus's expertise in programming doesn't necessarily translate to chip ISA design. And as such, Linus can get details (expected power-characteristics of future chips) dramatically wrong.
Consider also that Linus worked at Transmetta for a relatively long time trying to build a CPU company. I would trust his opinions on chip ISA design more than most software developers.
You’ve basically taken his post out of context, misinterpreted it to be saying something he isn’t, and claiming it’s wrong because some details have changed on the ground after he made it.
This is the problem with expert opinion. It’s so easy to misinterpret. It’s also possible to get some details wrong without invalidating the larger point. Figuring out when the expert is wrong is hard, but claiming that Linus doesn’t understand chip design just factually doesn’t understand the context of Linus’ perspective and his actual professional experience.
Quote the post in question:
> not with some AVX512 power virus that takes away top frequency (because people ended up using it for memcpy!)
This is pretty obviously talking about the down-clocking issue.
> You’ve basically taken his post out of context, misinterpreted it to be saying something he isn’t, and claiming it’s wrong because some details have changed on the ground after he made it.
AVX on Sandy Bridge was worthless: it wasn't any faster than SSE because Sandy Bridge was still 128-bit (even on AVX code).
Anyone who has followed ISA-development for decades would recognize that the first implementation of an ISA is often sub-par and sub-optimal. It wasn't until Haswell before AVX was faster than SSE in practice.
AVX512 was first implemented on Xeon Phi. Then it was ported over to servers in Skylake-X. AVX512 from there on remained a "1st generation" product for a few years (much like AVX on Sandy Bridge, before Haswell). From the history of these ISA-extensions, we generally can expect downclocking issues or inefficiencies in the first few implementations.
-----
In fact, some portions of AVX still remain suboptimal. vgather for example, isn't really much faster than standard load/stores. Maybe they'll be suboptimal forever, or maybe some hypothetical future CPU could optimize it.
Either way, I know that I shouldn't make an argument based on the "stupidity of vgather" instruction. There's an "obvious" optimization that could happen if that instruction becomes popular enough to deserve the transistors. At best, if I were to argue about vgather instructions, I'd preface it with "current implementations of vgather on Skylake-X", or similar wording.
CPUs change over time, much like programming languages or compilers. I expect a degree of forward-looking ability from experts, not for someone to get lost in the details of current implementations.
Finally, Linus’ main advice is to stop focusing on AVX and focus on architectural improvements. It took an enormous amount of engineers to implement AVX512 and then even more to get rid of the power hit. I trust Linus (and back that up with my own experience and observations in the industry) that Intel would have been far better served focusing on architectural improvements like Apple did. It’s a blind spot Intel has because they know how to bump clocks and create x86 extensions but they don’t trust architectural improvements to deliver massive gains. Hell, this isn’t the first time. Netburst (which was their attempt) screwed up so badly they had to return to the P3 architecture with the Core line. Intel has the engineers but their management structure must be so bloated an inefficient to fail with such regularity.
If I, 5-years or 10-years ago, claimed ARM sucks because its in-order, then I'd be making a similar mistake as Linus did. There's nothing about the ARM-ISA that forced in-order execution, it was just the norm (because back then: it was considered more power-efficient to make in-order chips).
There's nothing part of AVX512 that says "You must use more power than other instructions". That was just how it was implemented in Skylake-X. And lo-and-behold, one microarchitecture later, the power-issue is solved and a non-issue.
Similarly, ARM chips over the past decade switched to out-of-order style for higher performance. Anyone who was claiming that ARM was innately in-order would have been proven wrong.
----------
Maybe it takes experience to see the ebbs and flows of the processor world. But... that's what I'd expect an "expert" to be. Someone who can kind of look forward through that.
I mean, I've watched NVidia GPUs go superscalar (executing INT + FP simultaneously). I've watched ARM go from largely in-order microcontrollers into out-of-order laptop-class processors. I've watched instruction sets come and go with the winds.
------
If that's not good enough for you, then remember that Centaur implemented AVX512 WITHOUT power-scaling issues back in 2019, still predating Linus's rant by months. A chip did exist without AVX512 power throttling, and he still ranted in that manner.
If you had said 10 years ago "ARM sucks because it's in order" you wouldn't be factually wrong on that front in that in-order sucks for performance, but you would be factually wrong because the first iPhone was already partially out-of-order and 10 years ago you'd be looking at the iPhone 4 which was a Cortex-A8 superscalar design. Also in-order is fantastic for battery - that's why M4 & below are still in-order. Note that me saying this doesn't imply that will remain the case for all time, which is how you seem to be interpreting statements like that which would be an extremely poor interpretation on the part of the reader.
(TIL that my git repository at work is corrupt. Again. Second time in two years. "git fsck" thinks it's fine. "git repair" spins forever. Thinking hard about Fossil just now...)
We might reasonably guess that he is dead wrong about the merits of C++ as an implementation language. Observationally, he would have plenty of company.
Unfortunately, experts are only expert in a few subjects, and experts (well... all humans even) are bad at knowing when they've left their field of expertise.
Linus Torvalds for example is clearly an expert programmer. And then suddenly he talks about AVX512 and I wonder if he knows anything about low-level programming or optimization. (https://www.realworldtech.com/forum/?threadid=193189&curpost...).
Case in point: with Rocket lake (on 14nm nonetheless), the power-limits associated with AVX512 have already been largely lifted. It didn't even take a few months before Linus's words became obsolete.
https://travisdowns.github.io/blog/2020/08/19/icl-avx512-fre...
100MHz difference single-core, 0MHz difference multicore. Virtually no penalty for using AVX512 on new systems. Linus was ultimately trapped in the Skylake-X way of thinking, and not looking forward to the next chips.