News for hackers. Stuff that matters.
Re:Ask HN: Affordable hardware for running local large langu... by angoragoats on Sunday May 05, 2024 @01:06PM (#40266374)
This might technically be outside your budget, but if you happen to have a PC, I highly recommend the RTX 4060 Ti 16GB ($450, or less if on sale). It can easily handle 13B models and is quite fast. You don’t need a fancy PC to put it into; anything with a spare PCIe slot and a reasonable sized power supply will work.

These cards can easily be found at MSRP because they’re not a great improvement over the 3060/4060 8GB for gaming, but the added memory makes them excellent for AI.

There are no comments on this comment yet.