News for hackers. Stuff that matters.
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by chaxor on Thursday May 11, 2023 @04:44PM (#35908118)
They don't only run on high end systems. Good models can run on a desktop you have at home. If you don't have a desktop... I'm not sure what you're doing on HN.
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by throwawayadvsec on Friday May 12, 2023 @02:09AM (#35912572)
You have a weird definition of "good model"

Llama 7B is NOT a good model.

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by chaxor on Friday May 12, 2023 @12:12PM (#35918220)
You can run much larger models than llama-7B. Galpaca-30b or Galactica-120b for example.
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by int_19h on Friday May 12, 2023 @05:11PM (#35922467)
30B is still not good enough.

What kind of desktop are you running a 120B model on with reasonable performance?

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by chaxor on Friday May 12, 2023 @06:29PM (#35923323)
I would disagree that 30B is not good enough. It heavily depends on which model, and what you're trying to use it for.

30B is plenty if you have a local DB of all of your files and wiki/stackechange/other important databases places in a embedding vectordb.

This is typically what is done when people make these models for their home, and it works quite well while saving a ton of money.

While llama-7B systems on their own may not be able to construct a novel ML algorithm to discover a new analytical expression via symbolic regression for many-body physics, you can still get a great linguistic interface with them to a world of data.

You're not thinking like a real software engineer here - there are a lot of great ways to use this semantic compression tool.

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by int_19h on Friday May 12, 2023 @09:39PM (#35924617)
If I just wanted a fuzzy search engine for local data, I'd use a vector DB - there's no need for an LLM on top of that.