Re:AI chip startup Wave to buy Silicon Valley old-timer MIPS
by jcranmer on Thursday June 14, 2018 @05:55PM
(#17315772)
> CPU performance increases since 2000 have mainly come from longer pipelines, bigger branch prediction logic and larger/deeper caches. Those are all extremely important for single-threaded performance but are not much use for embarrassingly parallel (low branch) computation like in MATLAB/Octave, R, shaders, AI, physics and so on.
And better misaligned memory handling (Haswell has no latency penalty reading a word that crosses cache lines), better speculative execution, wider SIMD units, more complex SIMD swizzling operations.
> That's the part that I'm getting tired of. Companies tout the power of their various technologies without really addressing the hard problems in computer science. I want a paradigm that presents itself as a single unified CPU and memory. I want the compiler and operating system to optimize my code and handle moving my data with copy-on-write. If I had something like this and a decent Actor Model language, then it could trivially emulate things like shaders and we could get rid of all of these domain specific languages.
Speaking as someone who writes an optimizing HPC compiler... the idea that you can write a simple program and have it magically parallelize to ultra-high-throughput code is fantasy. We had that idea back in the 1960s. We've still not solved it 50 years later, and we've learned that there are giant barriers towards making these fantasy compilers that are rather fundamental.
One major problem you have to consider: communication is expensive. Already, on a single chip, it's impossible for data to move from one end of the chip to the other within a clock cycle (the speed of light in a vacuum takes about 10 times the clock cycle time to do so, and the speed of light in a physical wire is slower than that in a vacuum). So you start paying latency costs to have to move bits of data around. But your interconnect mesh isn't all-to-all, not when you have hundreds of access points. So that means that you have to pay attention to who is communicating to whom, and how much they are doing so. In a pathological case, like reversing a list, you've got all of your data moving across a single link in a 1-D network or only sqrt(#processor size) links in a 2-D network.
And better misaligned memory handling (Haswell has no latency penalty reading a word that crosses cache lines), better speculative execution, wider SIMD units, more complex SIMD swizzling operations.
> That's the part that I'm getting tired of. Companies tout the power of their various technologies without really addressing the hard problems in computer science. I want a paradigm that presents itself as a single unified CPU and memory. I want the compiler and operating system to optimize my code and handle moving my data with copy-on-write. If I had something like this and a decent Actor Model language, then it could trivially emulate things like shaders and we could get rid of all of these domain specific languages.
Speaking as someone who writes an optimizing HPC compiler... the idea that you can write a simple program and have it magically parallelize to ultra-high-throughput code is fantasy. We had that idea back in the 1960s. We've still not solved it 50 years later, and we've learned that there are giant barriers towards making these fantasy compilers that are rather fundamental.
One major problem you have to consider: communication is expensive. Already, on a single chip, it's impossible for data to move from one end of the chip to the other within a clock cycle (the speed of light in a vacuum takes about 10 times the clock cycle time to do so, and the speed of light in a physical wire is slower than that in a vacuum). So you start paying latency costs to have to move bits of data around. But your interconnect mesh isn't all-to-all, not when you have hundreds of access points. So that means that you have to pay attention to who is communicating to whom, and how much they are doing so. In a pathological case, like reversing a list, you've got all of your data moving across a single link in a 1-D network or only sqrt(#processor size) links in a 2-D network.