We've been building the wrong hardware for ML for a while now.

A TPU doesn't delivery supremacy for problems over GPU hardware. It's a Google senior engineering retention and PR project.

Exceedingly few problems resemble image recognition. You wouldn't be able to tell from the research, because due to the tools it's sort of the only affordable thing to do.

Playing Go barely looks like it. Or more accurately, Go doesn't look like most other games. Molecular physics and synthesis barely looks like it. Neural programs/neural Turing machines that matter (converting complex sim code like game rules and physics into neural networks via learning) may never achieve the accuracy or performance/watt necessary to really compete with just making a bunch of CPUs or straight up dedicated hardware.

Word embeddings are a disaster. People keep trying to do innovative stuff with it, and it's 2018 and we've just just recently got entity names from Wikipedia. I think Peter Norvig, having basically only machine translation to point to, is really going to eat crow when non-perceptual AI tasks sort of have nothing to do with stuff Google has developed. But what do I know.

I'm sure someone's going to trot out some obscure TPU or GPU-driven thing. It's been years! Awesome innovations in hardware really do lead to obvious, immediate crazy cool stuff. I'm really talking about the Kinect as an example here, in that it was really hyped and not much came of it. The TPU, and ML-specified hardware targeted to today's problems, is a slower-motion Kinect of our time.

Mining cryptocurrency, the most recent innovation, is the opposite of cool.

I'm confident these tools exist to capitalize on the subsidy from the gaming industry. AMD is wise to not chase ML features.

The underlying technology was acquired by Apple and is used on iPhones for FaceID, https://www.theverge.com/circuitbreaker/2017/9/17/16315510/i...

> PrimeSense was never solely responsible for the technology in Microsoft’s Kinect — as evidenced by the huge improvements Microsoft made to Kinect 2.0 on its own — and it’s also obvious that Apple is doing plenty of new software and processing work on top of this hardware. But the basic idea of the Kinect is unchanged. And now it’s in a tiny notch on the front of a $999 iPhone.

I'm not sure what you mean by "ML features", but AMD/ATI's dismissive attitude towards the ML and GPU-compute communities is looking quite silly by now.

I disagree, ML computing has tons of competition and its margins will be quickly diminish. I think they are right to focus on gaming.

It makes it extremely difficult to take any of what you just said seriously knowing that you made this statement.

People with academic backgrounds would be less afraid to point out how much vaporware machine learning has produced. They'd take stock in that their tenure, or six year PhDs, give them an opportunity to not chase trends so aggressively. They'd shill online less, for some giant faceless corporation like Google.

I'm not going to shit on a bunch of great stuff people have made. But the scary thing is that Noam Chomsky is probably right, that AI science hasn't advanced much at all. That we're still in an AI winter where it matters, the science, despite all the money and attention.

Do you mind clarifying what exactly your issues are with these specific applications of machine learning? Is it that you think there isn't really any meaningful value that can be gained in working within these areas? Or is it just that you want people to spend more time researching a different topic that you think would be more interesting/compelling?

This is coming from a physicist starting to investigate ML tools in my own work.

Its overall design sits somewhere between a GPU and CPU. Like a GPU, it supports 8-way hyperthreading (GPUs typically support 10. CPUs usually only support 2... maybe 4 for Larabee). Like a GPU, its programmed through an OpenCL-like language (I know you don't like that, but its how it works).

However, the overall layout is very CPU-like as well: a "City" is composed of "Villages", which is composed of multiple "Processing Elements". Each PE is scalar like a normal CPU, although 8-way SMT. So its not like a GPU's SIMD or SIMT architecture in this regards.

--------

See information here:

https://en.wikichip.org/wiki/pezy/pezy-scx/pezy-sc

It really is an interesting design that toes the line between CPU and GPU.

/s

Real talk, though. Why do you think larabee is the "right" hardware?

x86 isn't the right choice though. As the chips get smaller, all the x86 cruft starts to become a larger portion of each core.

RISCV with 40-ish instructions plus a SIMD extension would probably be ideal. They've already proven that those designs use less space per core than ARM.

I've also noticed that a lot of the time, people are comparing GPU to naive implementations (and sometimes, implementations written in dynamic languages) instead of to highly tuned BLAS or MKL implementations. For a large swathe of problem types, using the fastest math library will reduce the CPU/GPU gap to less than an order of magnitude or less.

https://software.intel.com/en-us/mkl/features/benchmarks

In no case will you see it get close to 10TFLOPS. GPUs easily do this, and can approach 100TFLOPS with tensor cores.

In your experience, are these comparisons using naive GPU implementations as well? If so, that still seems like a valuable comparison.

If they don't gimp FP64 it won't leave a whole lot of reason to buy the MI50 for probably 10x the price.

MI50 has ECC RAM support. That's absolutely necessary for high-end scientific compute.

It's not gimped according to Ryan Smith from AnandTech: https://twitter.com/RyanSmithAT/status/1083959608371175424

Which means it destroys it's only real competitor for consumer FP64 - the Titan V - which costs 3000 Dollar.

Since AMD will sell the Radeon VII directly at first, I think early adopters will probably face limited quantities rather than price markups (not counting unauthorized resellers). After that, we'll have to see how the situation with retailers and later on non-reference models shakes out.

https://news.ycombinator.com/item?id=18868617