News for hackers. Stuff that matters.

A comment on Apple's AI Strategy in a Nutshell. Parent comment

Re:Apple's AI Strategy in a Nutshell by elicksaur on Tuesday June 11, 2024 @09:02AM (#40645741)
If you stream the answer, the first token time is roughly the per token time.
[ Parent | on HN ]
Re:Apple's AI Strategy in a Nutshell by lostmsu on Tuesday June 11, 2024 @07:32PM (#40652882)
No, you have to feed the entire prompt token-by-token before getting response.
[ Parent | on HN ]
Re:Apple's AI Strategy in a Nutshell by elicksaur on Wednesday June 12, 2024 @12:04AM (#40654381)
Oh, I see, I misread. Thought it meant 0.6ms per output token. Now I get that it’s saying “prompt token”, so if your prompt is 100 tokens, that’s 60ms.

That seems pretty fast. 1.6k tokens for a 1s time. Do other models compare to that? I’m not sure what the current top ranking for this metric looks like.

[ Parent | on HN ]
Re:Apple's AI Strategy in a Nutshell by lostmsu on Wednesday June 12, 2024 @12:15AM (#40654438)
Latency here is weird. You are using the number as bandwidth in this calculation. Perhaps reporter doesn't really know what he's talking about.
[ Parent | on HN ]