News for hackers. Stuff that matters.

A comment on Claude just slashed the cost of building AI applications. Parent comment

Re:Claude just slashed the cost of building AI applications by hansonw on Sunday August 18, 2024 @07:25PM (#41286396)
It probably actually is several GB / 1M tokens. Pretty conservatively, if the KV embedding dimension for each token is ~10K x 100 attention layers (this is roughly the scale of Llama3.1 405B) that’s already 1M 16-bit floats per token = 2GB
[ Parent | on HN ]

There are no comments on this comment yet.