I've seen reports of people being able to run LLMs at decent speeds on old Nvidia P40s. These are 24GB Pascal GPUs and can be bought for as low as $100 (although more commonly $200) on eBay.

Link to report please

You can do it on CPU now.

Benchmarks: https://github.com/ggerganov/llama.cpp/issues/34

It should work. After the initial prompt processing, token generation is typically limited by memory bandwidth more than by compute.

Intel Larrabee has come into the chat but showed up about decade and a half too early.