To use an analogy: I'm not asking how far a car can go on the highway if accelerating from zero, but how far it can move if moving at highway speeds.
Nonetheless, the latency to main memory is also a great example of how bad programmers' intuitions are! A processor can perform thousands of operations in the time it takes for it to wait for just one memory access.
RAM is the new disk, and disk is the new tape.
PS: There was an article on here just a couple of days ago about how "handles are better than references". I didn't see many comments pointing out that this doubles the number of memory requests required when chasing pointers, halving the overall performance!
> To use an analogy: I'm not asking how far a car can go on the highway if accelerating from zero, but how far it can move if moving at highway speeds.
The terms and conditions keep applying. The everyday real world conditions implied by "highway speed" include regular cache misses and branches.
It's not about "start" or "acceleration". This massively impacts the steady state speed.
If you want a car analogy, tight vector packing compares to normal code about as well as looking at the speed and width of the flame front across a cylinder and using that to measure fuel consumption.
Again, terms and conditions.
The slow part for memory access isn't how much memory can be piped over the wire but rather how fast a request for the next chunk of memory can be issued. So while it's possible to get those 480 multiplications that relies on the cache being warm and the memory access being predictable to the CPU. If either of those two constraints are violated, then the number of multiplications is severely decreased.
In other words, the more frequently you have to tell the memory controller to get more memory, the less work you can do.
To put it in perspective, it costs around 100 cycles to load something from main memory. Imagine you are talking about something like an n body simulation that may be accessing memory all over the heap. In the worst case, you are looking at 4 multiplications (if multiplying by a scaler) and perhaps even just 2 multiplications if you need 2 memory pulls per multiplication.
Caches don't just help, they are necessary to come anywhere near achieving that 480 number.
Just a little history lesson.
Once upon a time, the memory controller didn't exist on the CPU, it was on a separate chip (the north bridge). Whenever the CPU wanted to load memory, it'd send a request to the north bridge and the north bridge would send that request to main memory.
One of the huge improvements to CPU performance for both Intel and AMD was integrating the memory controller onto the CPU. Why? Because they shortened the distance a request for memory, and the memory itself, had to travel significantly. Electrical signals travel at about 2/3 the speed of light and if your memory is 0.1 meter from the CPU, you can see how many requests for a new chunk of memory can really slow down everything.
This is fundamentally why nVidia's big GPGPU processors they sell to datacenters come chock full of memory. Because they want to limit the amount of time and times the GPGPU has to reach back into main memory to process stuff. It's far quicker to load up the 20gb of ram on the card in one go then to need to constantly pull that data over the PCIe line.