Yield problems hurt superlinearly as you scale up the size of a chip. That's why the fastest graphics cards have two GPUs on board, and why two mid-range cards will usually offer a better price/performance ratio than the biggest, fastest single-GPU card. Besides, the biggest GPUs out there are already little more than arrays of hundreds or thousands of vector processors: AMD's current biggest is 2.6B transistors divided among 1536 shader processors running at 800-900 Mhz, and NVidia's biggest is 3B transistors divided among 512 shader processors running at 1.5Ghz.

That NVidia monster has a die size of about 520 mm^2. At that size, there's already a lot of waste due to the fact that wafers are round and the chips are rectangular, and that can only be reduced by making physically smaller chips. (Rumor has it that by the time NVidia's 529mm^2 GF100 chip was originally supposed to launch, yields were bad enough that they were getting only about 2 usable chips per 300mm wafer. The cost of chip production has a pretty much linear relationship with the number of wafers processed, so that really hurt.)