> Computers today are closer to 10 times faster than computers of the 90s, not 10,000 times faster like they should be.

This isn't even close to true for many classes of problems.

For 3D rendering software running on CPUs, this is observably not true: todays machines are computing 10,000x faster than what was possible in 1994, when Toy Story was rendered. Not to mention massively larger data sets are being rendered now thanks to storage, memory, and general I/O improvements.

In fact, path tracing (now the dominant 3D rendering method, including at Pixar) was considered elegant but impractical in the 90s because it required way too many CPU cycles and way too much memory (the whole scene, roughly, needs to be in memory at all times). It took a 1000x (now 10,000x) increase in computer performance across all dimension to make path tracing viable.

Does any of that sound like just a roughly 10x improvement in 25 years?

I was excluding video cards, and comparing a 1999 computer (say a 300 MHz 1 core Intel Pentium II with 100 MHz RAM) to a 2019 computer (say a 3 GHz 8 core Intel I9 with 1 GHz RAM). There is some wiggle room there on clock and bus speeds and width, but all within a similar order of magnitude:

https://www.cpubenchmark.net/singleThread.html

This chart doesn’t go as low as we need, but we can extrapolate the Pentium II to a score of about 50 and the I9 to 2500 ( although if we include the I9’s 8 cores, we get a score more like 20,000 https://www.cpubenchmark.net/cpu.php?cpu=Intel+Core+i9-9900K... ). So we’re looking at about 50x faster single-threaded performance between 1999 and 2019.

That means that since 2019 clocks are 10x faster than 1999 clocks, there has been only a 5x performance increase per cycle (most of this probably due to longer pipelines, larger cache and wider busses which are evolutionary rather than revolutionary technologies).

Yes, there have been major price decreases. But only minor increases in performance. Computers have exponentially more transistors, but only linearly better processing power and bus speed. Had performance kept up with the 100x increase in transistor count per decade (10,000x every 20 years), The I9 with 8 cores should have scored about 5010,0008 = 4,000,000. But it only scored 20,000, so it’s only reaching about half a percent of its potential performance.

We don’t hear any of this talked about much because it’s an inconvenient truth. But from my perspective, Moore’s law ended around 2000. If it was up to me, I would scrap the rat race towards incrementally faster single-threaded performance. One divided by half a percent indicates that chips today should have roughly 8*50 = 400 cores to adequately make use of their potential processing power. For a hardwired (non-FPGA) chipset, I’d reach higher than that and target 1024 cores running at 3 GHz, with as much as 16 GB of on-chip RAM running at 1 GHz (16 MB per core so they can be programmed with local copies of their own operating system), for a chip with roughly the same transistor count as today, and costing under $1000.

My training was for designing a MIPS-style chip running at roughly 100 MHz. I don’t see any mystery in designing the scaled-up chip I’m suggesting. In fact, it would be far simpler and cheaper to design because we’d only have to do it once for perhaps a 4-6 stage pipeline with little or no branch prediction or cache logic and then repeat that in a 2D mesh. A single core would run up to 5x slower than an I9’s, but we’d have over 1000 more of them to do whatever we wanted with.

In case I wasn't clear, the kind of 3D rendering I'm describing doesn't use graphics cards. It's all on the CPU.

As for ideas to make a faster CPU, sure, knock yourself out. But there's no question that computers today are much closer to 10,000x faster after 25 years of development on actual, real-world code being run to earn billions of dollars. Your approximately 10x faster claim after 25 year is just wrong, at least with the software systems I'm familiar with.

There is a typo in the above formatting that I can't edit:

5010,0008 = 4,000,000

Should say:

50 * 10,000 * 8 = 4,000,000

That was important to my central point that an I9 has roughly 400 times fewer cores than it would have had it kept up with Moore's law, so I didn't want to leave it unclear.

At the lower end of your range this sounds somewhat like a Xeon Phi - up to 72 cores at 1.5 GHz with 512 kb L2 cache per core. Intel has discontinued the Xeon Phi line, presumably in part because there were limited applications where such an architecture proved compelling. What advantages would you see your FPGA based architecture having over something like the Xeon Phi that might lead it to have more success?

That's a good question, and I'm honestly not sure. I think highly parallel, general purpose and customizable computers have uses outside the mainstream. So they would make great ray tracing video cards and platforms for physics simulations for science research, stuff like that.

I think most problems in computer science today can be traced to the original sin of choosing Von Neumann architecture and imperative programming over reprogrammable hardware and functional programming. That's why we think of things like machine learning as rather difficult problems, even though they are just simple algorithms that are compute-bound.

Basically my complaint is that this stuff has progressed in spite of our choices, not because of them. The proprietary and closed-source hardware, drivers and programming paradigms we have to deal with today have set us back at least 20 years. Probably more like 40 years if you include some of the shenanigans that happened between IBM, Intel, Microsoft and Apple in the 80s and 90s. They solved personal computer problems wonderfully, but inadvertently hindered the progress of computer science in general.

I don't have an answer for why the market tends to reject the stuff I talk about. I just know that mathematically it's very first-principles stuff that I really hoped would have caught on by now. To me it feels like these ideas keep getting sabotaged for economic and political reasons rather than scientific ones.

Edit: I forgot to mention that FPGAs solve this problem by providing generic reprogrammable hardware whose use can't be monopolized the way that single-purpose hardware is. Something like open source FPGAs (that don't burn up when they're programmed wrong) would allow us to explore many more problem spaces in computer science than we can currently.

Xeon Phi evolved from Larrabee which was Intel's attempt to make a GPU based around highly parallel general purpose x86 cores. Many smart people thought it was a good idea and worked on it but it failed. Xeon Phi in theory might be good for ray tracing but all the action in ray tracing acceleration today is with custom hardware added to traditional GPUs.

Intel poured a lot of resources into this approach and it didn't find success so I'm not really sure you can say it was sabotaged for economic or political reasons, rather it failed to find a market.

I think the reasons it failed are complex and perhaps it's an idea whose time has just yet to come rather than being a bad idea but it has been tried without much success more than once. I backed the Parallella Kickstarter and that didn't really go anywhere either.

Ya I actually bought an Nvidia GeForce RTX 2070, specifically to play around with its ray tracing capabilities. I kind of wish I could play around with a Google TensorFlow processor (TPU):

https://en.wikipedia.org/wiki/Tensor_processing_unit

It's an ok chip and has some of the ideas that I mentioned (like an 8 bit design that could possibly emulate 64 bit and higher via microcode or software) but I'm afraid that it's too narrowly scoped to be of much use in other domains. Companies seem to be exploring branches of the parallel/DSP problem space rather than providing general solutions like the Xeon Phi.

I'm not totally sold on high multicore (say 64 or more cores), but it's the only way that I can see forward since Moore's law ended. It would be much better to have increased single-threaded performance since that can emulate multicore. Multicore can't emulate single-threaded above Amdahl's limit of about 4x, which is why very few chips today have more than 4 or 8 cores.

That said, most of the problems that I find interesting are also embarrassingly parallel. And since there is currently no consumer high-multicore processor, I can't explore those problem spaces. From first principles, it should have gone:

1) Single core: 6502, z80, 286 2) Low multicore: Pentium, POWER 3) High multicore (MIMD): Xeon Phi, Parallella 4) Optimized vector processing (SIMD): GPU, TPU

Without a viable #3, we have no way of knowing if #4 is an optimal solution or an evolutionary dead end. We have to drink the kool-aid and proceed out of trust. That's the part that I find extremely disappointing about the last 20-25 years.

Another way to say that is: if there is no market fit for ever-faster processors at cheaper prices.. that consumers would rather have low-cost mobile processors and regressively rent time on data centers.. that they'd rather have photorealistic rasterized video games than true supercomputing at home, then computer science is at the very least wounded (and I would argue, slowly dying).

This is a nuanced issue that the mainstream isn't even aware of. I'm excited to see some people trying to get projects like Parallella off the ground. From what I can tell, they are one of the only chips working from first principles:

https://en.wikipedia.org/wiki/Adapteva

But as you pointed out, it may take more than enlightened engineering to bridge the gap to next-gen computing. That's the reason why we are only today seeing advances in machine learning that were fairly well understood by 2000. I just don't want to wait around another 20 years for the other good stuff to get here. Which is why I think it might take an open source design on inexpensive FPGA hardware to show people what they're missing.

> And since there is currently no consumer high-multicore processor, I can't explore those problem spaces.

It depends on your definition of "consumer" but you can buy a 32 core Threadripper at the top end of consumer now and there's a chance we might see 64 core Threadripper this year. They're not exactly cheap but they are likely cheaper and faster than anything comparable you could do with an FPGA and they are fully compatible with existing toolchains and operating systems.

If it's so trivial to write a high performance Lisp-to-LUT or Lisp-to-gates toolchain, why don't we have one?

We do! It's called Fairylog, and it was on HN last month.

https://news.ycombinator.com/item?id=19701279

Thank you very much! Somehow I missed that article, but have favorited it because it's such an important stepping stone towards real mainstream adoption of reprogrammable hardware.

>"* A Lisp to HDL compiler, preferably with optimization. [time-consuming, but not difficult]"

>"* A high-level functional or vector or stream language (Clojure/Elixir/MATLAB/Erlang/Go) to Lisp transpiler.

I am curious about the need to use LISP and a functional language for general purpose FPGA computing. Can you elaborate?

>"So we could play around with things like copy-on-write (COW), content-addressable memory and embarrassingly parallel computation in all the areas that today's computers are weak at (things like ray tracing, voxels, genetic programming, neural nets and so on)"

What is COW here? I am only familiar with it in context of a kernel optimization technique for child processes but I think you might be referring to something else here?

Oh ya, well, most programming today is akin to the macro programming that was done in the 80s in languages like Visual Basic. There is essentially no difference between that and the imperative languages like C++, Javascript, PHP, Ruby, Rust, Swift, etc that have captured the market.

VHDL and Verilog aren't really imperative languages. They're functional programming (FP) like spreadsheets, Lisp and languages built on FP like Elixir, Clojure, Julia, Haskell, etc but with ugly syntax.

Another way of saying that is that a digital circuit is analogous to a spreadsheet (which is FP). So we could use referential transparency and other analyses/optimizations from FP and apply them to circuit analysis like Thevenin and Norton equivalent circuits, but at scale.

Sorry the copy-on-write (COW) thing was a little obscure. What I was getting at there is that buzzwords like Map Reduce and big data are hand waving over the central concept of data locality. So for example, a CPU-RAM computer might store a database on an SSD and use a bunch of fancy algorithms to load, cache and process that data. But a multicore computer could slice up the database and store a percentage of the data locally by each core. That's what Map Reduce and Erlang/Go and the Actor model do.

At that point, we can use concepts from the web like content-addressable memory and have an abstraction that frees us from the micromanagement of caching things. Basically we wouldn't have to worry about finding the fast path anymore, because with 1000 cores, anything we do is fast. We'd just have to keep the cores fed with data.

IMHO this core issue is why multicore computers that connect to a single RAM have generally failed. We need multicore computers with data locality, so that each core kind of acts like it has its own OS or is a node in a cluster. That would allow us to build on supercomputer algorithms that are well-understood. Then with those basic building blocks, things like running genetic algorithms and protein folding and physics simulations become straightforward because we're able to use simple algorithms running in parallel rather than increasingly complex algorithms running serially on CPUs whose performance have stopped improving.

Thanks for the wonderfully detailed response and insights. I had no idea that RTL like VHDL/Verilog were FP.

Would you mind elaborating on the following which I found really interesting:

>"Oh ya, well, most programming today is akin to the macro programming that was done in the 80s in languages like Visual Basic. There is essentially no difference between that and the imperative languages like C++, Javascript, PHP, Ruby, Rust, Swift, etc that have captured the market."

I have no experience with VisualBasic so I think an important point your making is being lost on me. How is there no difference between VisualBasic and all those other disparate languages exactly? Thanks.

"I had no idea that RTL like VHDL/Verilog were FP."

They really aren't. I mean, not syntax-wise at least. Conceptually there are similarities, but really they are process programming, if anything. Processes are everywhere. Not OS processes, not OS threads, basically green threads. It's really asynchronous event-driven programming. Each processes models a distinct bit of hardware, with all the bits of hardware running in parallel. If you've done node or python's twisted then you are actually in a good place to understand. It's tricky though because there are several different language constructs for creating processes and it's all shared memory to communicate between them. If you aren't careful you don't always realize that you've created yet another process (the assign keyword tricks people). If you get outside the subset of the language that is synthesizable (able to be translated directly into hardware), like for simulations, you get to deal with race conditions and everything.