News for hackers. Stuff that matters.

A comment on Gradient AI Releases 1M Context Llama 8B. Parent comment

Re:Gradient AI Releases 1M Context Llama 8B by zackangelo on Monday April 29, 2024 @05:08PM (#40204147)
The table at the bottom says they initialized the 65K version from "LLaMA-3 7B"? (Assuming the 7B is a typo and they meant 8B.)

And each successive version with a larger window was initialized on the previous smaller one (65K -> 262K -> 524k -> 1048k).

[ Parent | on HN ]
Re:Gradient AI Releases 1M Context Llama 8B by forrestp on Monday April 29, 2024 @05:12PM (#40204188)
Right. We are sleep deprived -- couldn't stop over the weekend. Please forgive the typos
[ Parent | on HN ]