If you don't need good single thread performance and don't care about Performance/Watt that much you can get used servers for very little money with lot's of DDR3 memory (128GB+) and these X5670 als dual cpu systems. If you don't do anything fancy you get 80% of performance for 20% of the price of a new system.

Or PCIE performance since the PCIE is still on the chipset on those CPUs.

You can also buy used Xeon workstations and put in a new GPU card. The main problem is that the cooling isn't especially quiet, and they have proprietary motherboards and power supplies.

The PSUs are still essentially ATX, so that's usually not a big problem.

Form factor on the other hand... well. Sometimes they really go off the reservation there.

Not necessarily. I have a HP Z400 workstation with what appears to be an ATX PSU... but some of the pins are swapped and it won't actually boot with a non-OEM PSU. You can make an adapter, but in general it's not a safe assumption that this class of systems are compatible with off-the-shelf PSUs. These are OEM systems and the only thing that is guaranteed to work is OEM parts.

https://www.badcaps.net/forum/showthread.php?t=47754