News for hackers. Stuff that matters.
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Thursday March 28, 2024 @01:38PM (#39854914)
I ended up getting a 2nd hand 3090 for 680€.

Funnily, I think the card is new (smells new) and unused, most likely a scalper bought it and couldn't sell it.

Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by Zambyte on Friday March 29, 2024 @12:14AM (#39860626)
Nice, that's definitely a sweet deal
Re:Introducing DBRX: A New State-of-the-Art Open LLM by Data... by XCSme on Friday March 29, 2024 @12:23AM (#39860680)
Thanks, I chose a 3090 instead of 4070ti, it was around $200 cheaper and has 24GB vs 16GB VRAM and a similar performance. The only drawback is the 350W TDP.

I still struggle with the RAM issue on Ollama, where it uses 128GB/128GB RAM for Mixtral 24.6GB, even though Docker limit is set to 90GB.

Docker seems pretty buggy on Windows...