Re:Open models by OpenAI
by Rhubarrbb on Tuesday August 05, 2025 @02:39PM
(#44802338)
Generation is usually fast, but prompt processing is the main limitation with local agents. I also have a 128 GB M4 Max. How is the prompt processing on long prompts? processing the system prompt for Goose always takes quite a while for me. I haven't been able to download the 120B yet, but I'm looking to switch to either that or the GLM-4.5-Air for my main driver.
[ Parent
| on HN ]
Here's a sample of running the 120b model on Ollama with my MBP:
```
total duration: 1m14.16469975s
load duration: 56.678959ms
prompt eval count: 3921 token(s)
prompt eval duration: 10.791402416s
prompt eval rate: 363.34 tokens/s
eval count: 2479 token(s)
eval duration: 1m3.284597459s
eval rate: 39.17 tokens/s
```
Re:Open models by OpenAI
by anonymoushn on Tuesday August 05, 2025 @02:50PM
(#44802479)
it's odd that the result of this processing cannot be cached.
It can be and it is by most good processing frameworks.
You mentioned "on local agents". I've noticed this too. How do ChatGPT and the others get around this, and provide instant responses on long conversations?
Re:Open models by OpenAI
by bluecoconut on Tuesday August 05, 2025 @06:15PM
(#44805123)
Not getting around it, just benefiting from parallel compute / huge flops of GPUs. Fundamentally, it's just that prefill compute is itself highly parallel and HBM is just that much faster than LPDDR. Effectively H100s and B100s can chew through the prefill in under a second at ~50k token lengths, so the TTFT (Time to First Token) can feel amazingly fast.
Re:Open models by OpenAI
by mike_hearn on Wednesday August 06, 2025 @05:13AM
(#44809661)
They cache the intermediate data (KV cache).