Re:Hunting deep-seeded kernel bugs
by pmjordan on Wednesday March 28, 2012 @08:47AM
(#3765898)
I think it's a requirement of PCIe for devices to support DMA to memory with 64-bit physical addresses.[1] In any case, the hardware clearly supports it in the case - the bug was that a 32-bit I/O vector was being passed off to the hardware claiming to be a 64-bit one, and subsequently interpreted as such.
The situation was certainly bad with classic PCI - I have an nForce5-based motherboard whose PCI bus silently truncates 64-bit DMA buffer addresses to 32 bits. The onboard ethernet chip does in principle support 64-bit DMA, so I was getting all sorts of weird behaviour after upgrading my RAM from 2 to 6 GiB. The southbridge chip in question is now blacklisted for 64-bit DMA in the Linux kernel.
The 32-bit DMA cutoff isn't the only such constraint on buffer physical addresses of course; historically, ISA devices could only DMA to 20-bit addresses IIRC (or was it 24 bits?). I wrote an OS X driver for the virtio paravirtual ethernet adapter a while back[2] and its I/O buffers must be 4K page-aligned (low 12 bits 0), and its high bits are specified by a 32-bit number. This means you can only use buffers with physical addresses which fit in 44 bits. That's 16TiB, which are probably still a way off but could conceivably be reached not too far in the future. Current x86_64 CPUs certainly support 48 bit physical addresses in principle.
[1] That doesn't mean hardware bugs don't exist, but at least it's part of the basic spec now, as opposed to being tagged on later as for PCI.
The situation was certainly bad with classic PCI - I have an nForce5-based motherboard whose PCI bus silently truncates 64-bit DMA buffer addresses to 32 bits. The onboard ethernet chip does in principle support 64-bit DMA, so I was getting all sorts of weird behaviour after upgrading my RAM from 2 to 6 GiB. The southbridge chip in question is now blacklisted for 64-bit DMA in the Linux kernel.
The 32-bit DMA cutoff isn't the only such constraint on buffer physical addresses of course; historically, ISA devices could only DMA to 20-bit addresses IIRC (or was it 24 bits?). I wrote an OS X driver for the virtio paravirtual ethernet adapter a while back[2] and its I/O buffers must be 4K page-aligned (low 12 bits 0), and its high bits are specified by a 32-bit number. This means you can only use buffers with physical addresses which fit in 44 bits. That's 16TiB, which are probably still a way off but could conceivably be reached not too far in the future. Current x86_64 CPUs certainly support 48 bit physical addresses in principle.
[1] That doesn't mean hardware bugs don't exist, but at least it's part of the basic spec now, as opposed to being tagged on later as for PCI.
[2] http://github.com/pmj/virtio-net-osx