News for hackers. Stuff that matters.

A comment on Anthropic - Introducing 100K Token Context Windows, Around 75,000 Words. Parent comment

Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by nico on Friday May 12, 2023 @10:06AM (#35916560)
How much would they cost aprox per training?

What would the quality of the model be, compared to what Karpathy uses on his video “let’s build gpt from scratch”?

In that video, he builds a decoder-only transformer model, that learns Shakespeare from 1MB of data and trains in 15min

[ Parent | on HN ]
Re:Anthropic - Introducing 100K Token Context Windows, Aroun... by jiggawatts on Sunday May 14, 2023 @12:51AM (#35935411)
These just fine-tune existing models, so the cost is single digit dollars. Whatever it costs in electricity to run a desktop GPU for a day or a few days.
[ Parent | on HN ]