News for hackers. Stuff that matters.
Re:AI's Dial-Up Era by loyalcinnamon on Tuesday November 04, 2025 @02:51AM (#45808466)
As the hype is dying down it's becoming a little bit clearer that AI isn't like blockchain and might be actually useful (for non generative purposes at least)

I'm curious what counts as a midsize model; 4B, 8B, or something larger/smaller?

What models would you recommend? I have 12GB of vram so anything larger than 8B might be really slow, but i am not sure

Re:AI's Dial-Up Era by DSingularity on Tuesday November 04, 2025 @10:05AM (#45811815)
It can depend on your use case. Are you editing a large code base and will thus make lots of completion requests with large contexts?
Re:AI's Dial-Up Era by riskable on Tuesday November 04, 2025 @03:18PM (#45815429)
My take:

Large: Requires >128GB VRAM

Medium: 32-128GB VRAM

Small: 16GB VRAM

Micro: Runs on a microcontroller or GPUs with just 4GB of VRAM

There's really nothing worthwhile for general use cases that runs in under 16GB (from my testing) except a grammar-checking model that I can't remember the name of at the moment.

gpt-oss:20b runs on 16GB of VRAM and it's actually quite good (for coding, at least)! Especially with Python.

Prediction: The day that your average gaming PC comes with 128GB of VRAM is the day developers will stop bothering with cloud-based AI services. gpt-oss:120b is nearly as good as gpt5 and we're still at the beginning of the AI revolution.