News for hackers. Stuff that matters.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by Catloafdev on Friday July 31, 2026 @03:52PM (#49127833)
Neat! But, what do you do with a 0.5tk/s LLM?

Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?

Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by gcampos on Friday July 31, 2026 @05:05PM (#49128509)
You could use it for long run tasks while you don’t use the laptop.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by ElectricalUnion on Friday July 31, 2026 @06:22PM (#49129194)
By the time the tokens start coming out 30h later you might need to use your laptop again...
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by dotancohen on Friday July 31, 2026 @05:19PM (#49128648)
Have it summarise the week overnight for the meeting in the morning. Then have it summarise the meeting transcription overnight for the report tomorrow. Then someone else will have it summarise the report overnight to read on a 6" handheld screen in the small office the next morning after breakfast.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by ElectricalUnion on Friday July 31, 2026 @06:23PM (#49129206)
Then have it summarise the meeting transcription over the entire week for the report next week.

ftfy.

Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by Dylan16807 on Friday July 31, 2026 @10:32PM (#49130546)
If we estimate a meeting with pauses between speakers as 2.25 words per second, and .75 words per token, then a meeting generates 3 tokens per second. This says prefill and decode are both .5 tokens per second? Then each hour of meeting turns into 6 hours to read and 1 hour to output a summary. You could summarize two hours of meeting overnight, not too bad.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by mh- on Friday July 31, 2026 @11:53PM (#49130870)
Using half a kilowatt-hour, and if thinking is disabled during inference, yes.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by ElectricalUnion on Sunday August 02, 2026 @03:06PM (#49147329)
Aren't those speeds for the first few tokens, that, because of no context for attention to attend, are much faster to compute that the others? I expect the actual token speed to nosedive sharply as you get more context utilization.

What about the system prompt you're using? General purpose harness like Claude Code will insert their happy 22k tokens, even before your first useful token is processed. That might make it a no-go even _before_ you can even start, as the maximum context for this seems pretty limited (to make it fast)

And those LLMs are all "thinking", that is, rather that "one-shoting" the answer, they generate a lot of internal use reasoning tokens before starting to generate useful, visible response tokens. You can easily get to 30k tokens when your initial prompt is vague ambiguous garbage (as are naive transcriptions) as your LLM will "But wait, the user might have meant X, let me think more about this" lots of times.

No thinking (therefore much worse answers) will be a requirement.

Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by Dylan16807 on Sunday August 02, 2026 @06:26PM (#49149054)
I have no idea how this thing performs over time. But you don't need to give it a fancy prompt to summarize, and you could cache the prompted state too.

Yes I'm assuming you one-shot the summary or close to it. A summary doesn't need a zillion tokens of extra processing. I doubt it will be "much worse" for this use case, but I admit I don't know how low you can turn the thinking.

Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by throwawayffffas on Saturday August 01, 2026 @02:49AM (#49131783)
You get 8 nvmes set them up in raid 0/1 across two full pcie5x16 ports and you could reach up to 4ish tokens per second, presumably.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by withinboredom on Saturday August 01, 2026 @09:53AM (#49134478)
The problem is dram bandwidth to the cpu. Each token costs roughly 20gb of traffic and ddr5 is roughly 50-80gb/s, plus you still have to run the compute sequentially. That’s your limit.
Re:Run Kimi K3 using 29 GB of RAM at 0.50 tok/s by jurgenburgen on Saturday August 01, 2026 @10:06AM (#49134573)
> Neat! But, what do you do with a 0.5tk/s LLM?

Hopefully resolve incidents faster without people pasting slop into the incident thread.