Furthermore, the high-performance DDR3+ controllers typically hash the data word with the address so that when a repetitive data stream is transmitted it doesn't generate more EMI. (Some controllers also hash with a random seed gaining resilience against chilling the DIMMs of a running machine and reading them out on another machine in search of sensitive data.)
I find it really cool that any time you change the DIMMs in your computer, it essentially has to measure the length of the wires to the ICs on it. (I've found it less cool to have to manipulate timing values to compensate for deviation from PCB design rules, but thankful that it's possible. The fun of board bring-up.) If your BIOS has a "fast boot" option, mostly that means it remembers the wire lengths from last time so it doesn't have to do the measurement again every boot.
But now, sheesh! You need to carefully constrain the PCB CAD program so that all the lines match to within 0.1" or less. And, as you mention, that's just the tip of the design iceberg.
It's no longer possible to layout computers at low cost in a garage. Oh, well. Now hipsters sit around in open offices in SOHO and create silly apps.
Notice the reason why! In fact, I think the idea of a startup producing laptops targeted at developers has been mentioned before here on HN.
I'm in a chatty mood tonight, so I hope you'll excuse my indulgence of another few 'graphs on modern memory bus stuff I think is cool that I had to learn under product ship-date duress: Dynamic Termination.
Remember the high-school physics demonstration of the "reflected wave" where you grab one end of a rope that's tied off at the other end, give it a flick, watch the wave travel from your hand to the knot, then reflect back? Well kind of the same thing happens with voltage level on high speed data signals.
If your high school was fortunate enough in these (or those) fiscally troubled times to afford the equipment, the next demonstration tied off the same rope to some sort of damping spring that absorbed your wave's energy and didn't reflect it back at you. Those springs translate to "termination impedance" in electrical circuits.
So in order to avoid reflections, the address/control/clock lines of the memory bus "fly by" each and every DRAM IC until they reach the end of their path, where they "terminate" via resistors that prevent reflection to a voltage potential (half-way between 1- or 0- value potentials so that it's no more work to drive a 1 that 0 or vice-versa).
The data lines are wired directly from the memory controller to the DRAM chips so the DRAM chips have termination built-in to absorb the energy at the end of the line and prevent reflections.
Except that maybe there's more than one DIMM. DDR3 chips let you command them to connect or disconnect their termination resistors. So when you are accessing the closer-to-the-controller DIMM, the next one that's further out can apply termination. And when addressing the one that's furthest out, you disable on-die termination and rely on the actual resistor on the motherboard at the end of the bus.
My hard-learned lesson: the signals don't so so well when you get confused about which rank of chips are closer and which are further away.
I remember reading about the PCI bus, which I think is the first time I heard of a mainstream digital system requiring serious transmission line design.
Address pins are another matter. The Z80 had a built in static RAM refresh circuit, and there is some schtick about which addresses are refreshed as a group (rows or columns, I forget which). So, rearranging the address bus might result in some nasty surprises. And it might get even more interesting with more modern memory devices, which are way over my head.
On a whim, I got the Howard Sams book on the Z80 while I was in high school, around 1981, and I devoured it.