News for hackers. Stuff that matters.
Re:Kimi-K3 Technical Report [pdf] by GodelNumbering on Monday July 27, 2026 @12:41PM (#49072173)
Back of the envelope calculation (could be off, correct me if I am)

If you are a large enough company that spends million+ on inference a month, it makes sense to buy a GB300 rack ($6M on top range from what I could find) which has 20.7 TB. Since the model is mixed trained (MXFP4), you would need less than 10% of the rack's memory to serve the full model. Aggregate HBM bandwidth: 576 TB/s. You can run over 6000 parallel agentic workflows (each with ~100k context on average) at ~30 tok/s.

Assuming the annual amortization+electricity at $1.5M/year and about 50% average annual utilization, you get less than 60 cents (USD) per million output token, for a frontier model with plenty of capacity to share, all your data never leaving premises and well over an order of magnitude cheaper!

As long as a company believes that the openweight models will continue to get more capable and 'AI is here to stay', this model provides the first solid footing for a decision to just buy a rack.

Re:Kimi-K3 Technical Report [pdf] by FuriouslyAdrift on Monday July 27, 2026 @02:36PM (#49073838)
We're running Kimi 2.8 on a $107k server and getting around 50k tokens per second or better on most things.

We've already saved money compared to last years token cost on Claude/Gemini

Re:Kimi-K3 Technical Report [pdf] by logicallee on Monday July 27, 2026 @04:18PM (#49075019)
Are you developing software? Is most of it used on a coding agent? (Like Claude Code or ChatGPT Codex?) If so, what coding agent do you use? If you're not developing software what do you use it for (roughly)?
Re:Kimi-K3 Technical Report [pdf] by FuriouslyAdrift on Tuesday July 28, 2026 @05:11PM (#49089959)
Our use cases are data analysis, software development, general AI chat with RAG so far.
Re:Kimi-K3 Technical Report [pdf] by mdp2021 on Monday July 27, 2026 @03:57PM (#49074808)
> We're running Kimi 2.8 on a $107k server

Equipped with what? Is it CPU based inference, a mix...?

Re:Kimi-K3 Technical Report [pdf] by wongarsu on Monday July 27, 2026 @05:01PM (#49075477)
Not GP, but my educated guess is that they are running a system with between 4 and 6 MI325 or MI355x or similar AMD GPUs. With the 50k tps as the total figure for all parallel requests. Those cards have a lot of memory for their price, allowing you to push to really high batch sizes while still having a large context size for each request
Re:Kimi-K3 Technical Report [pdf] by FuriouslyAdrift on Tuesday July 28, 2026 @05:10PM (#49089952)
4x MI300A in a Gigabyte server
Re:Kimi-K3 Technical Report [pdf] by freediddy on Monday July 27, 2026 @04:21PM (#49075042)
how many requests per second can the server take?
Re:Kimi-K3 Technical Report [pdf] by FuriouslyAdrift on Tuesday July 28, 2026 @05:13PM (#49089979)
That's about it across the whole server. Kimi 2.8 is about as big as we can go. Kimi 3 is massive.
Re:Kimi-K3 Technical Report [pdf] by ajju on Monday July 27, 2026 @04:24PM (#49075077)
What is your config?
Re:Kimi-K3 Technical Report [pdf] by retinaros on Tuesday July 28, 2026 @10:14AM (#49084214)
on 1 paralel request maybe
Re:Kimi-K3 Technical Report [pdf] by FuriouslyAdrift on Tuesday July 28, 2026 @05:12PM (#49089972)
Correct but even with 30 simultaneous users, we're not seeing any noticable lag compared to AIaaS even with agentic workloads.
Re:Kimi-K3 Technical Report [pdf] by 999900000999 on Monday July 27, 2026 @12:52PM (#49072368)
And hire 2 or 3 dev ops to keep it running ?

That another 400 to 700k.

It becomes your problem and not someone else’s. However, I don’t trust hosted LLMs for anything that needs to be private.

Re:Kimi-K3 Technical Report [pdf] by wongarsu on Monday July 27, 2026 @12:57PM (#49072439)
Where do I sign up to get 200k/yr to keep one rack running? Sounds like an incredibly chill job
Re:Kimi-K3 Technical Report [pdf] by arjie on Monday July 27, 2026 @01:00PM (#49072490)
Apparently it’s going to take the 3 of us to do this, mate. Going to get so much reading done.
Re:Kimi-K3 Technical Report [pdf] by LeonM on Monday July 27, 2026 @01:20PM (#49072757)
What you get is not what you cost.

40% overhead is quite typical, so you'd be looking at $120k/year. In the USA I'd consider that a competitive salary for an admin capable or keeping a $6M rack of specialized hardware running 24/7.

Re:Kimi-K3 Technical Report [pdf] by margalabargala on Monday July 27, 2026 @02:00PM (#49073314)
Yeah but you don't need two such people, or even one, dedicated to this single rack.

A company of the size that this is worthwhile for, probably has dedicated devops on staff already and can add this rack to the inventory with no additional staff.

Re:Kimi-K3 Technical Report [pdf] by 999900000999 on Monday July 27, 2026 @02:05PM (#49073374)
Who is going to upgrade the models ?

Who is going to fix it when the api does something weird ?

Who is going to proactively make sure it’s not overheating?

Chat GPT has enterprise contracts for a reason.

Re:Kimi-K3 Technical Report [pdf] by margalabargala on Monday July 27, 2026 @03:11PM (#49074306)
The same person managing the company's email accounts and whatnot.

I didn't say it's fire-and-forget. I'm saying all that is maybe a day of work every 3 months.

Re:Kimi-K3 Technical Report [pdf] by DiskoHexyl on Monday July 27, 2026 @05:46PM (#49075921)
It doesn’t really work like that.

The companies which have the will and the budget to host their own LLMs typically require a ton of other, much smaller, models as well. They have internal security requirements, guardrails, audit, critical workflows start depending on your onprem setup, downtime is now something that’s not even allowed. There’s going to be a zoo of tooling, lots of bespoke work with internal clients who have no clue about docker, but now want their vibe-coded app to access a model they downloaded yesterday, and this model better be served and monitored 24/7 because now C-levels use it.

No one is going to budget several millions to then look at an email admin who hears ‘cuda’ for the first time in their life and ask them to just support the entire thing somehow.

And god forbid it’s an AMD setup.

This obviously isn’t relevant for a 10-person startup and their second-hand xeon with a single H100.

From my experience only the companies which are REALLY interested in privacy and data security bother with hosting their own models

Re:Kimi-K3 Technical Report [pdf] by margalabargala on Monday July 27, 2026 @09:22PM (#49078079)
That's some of those companies not all of them.

I agree that what you've described is probably worth a dedicated person. The earlier comment describing one rack running one model, is not.

Re:Kimi-K3 Technical Report [pdf] by stymaar on Monday July 27, 2026 @03:02PM (#49074190)
All of the answers to your questions above are in gp's comment already:

> A company of the size that this is worthwhile for, probably has dedicated devops on staff already

Re:Kimi-K3 Technical Report [pdf] by niltecedu on Monday July 27, 2026 @04:42PM (#49075258)
My company is not even the same ballpark, but even we already have people for it. And doesn't include the fact that you can get a colo location and just a MSP or a contractor to do it for you
Re:Kimi-K3 Technical Report [pdf] by w0m on Monday July 27, 2026 @03:53PM (#49074760)
> I don’t trust hosted LLMs for anything that needs to be private

I'd update this to

'I don’t LLMs for anything that needs to be private'

What's to prevent the LLM from sliding a heavily obfuscated binary blob into the application that does nefarious things? If you aren't creating the LLM itself from scratch, I don't feel it can be trusted.

Re:Kimi-K3 Technical Report [pdf] by JacobAsmuth on Monday July 27, 2026 @04:00PM (#49074840)
Why do you feel that creating the LLM from scratch is sufficient to trust it? Are you suggesting that you personally would read all 15 trillion tokens (plus every single agentic trade used in RL, along with its relative advantage in the batch) and personally guarantee that gradient descent would train a model which would not exfiltrate your corporate data?

Or that perhaps you have a perfect alignment algorithm which you are unwilling to share with the broader research community (evil)?

Re:Kimi-K3 Technical Report [pdf] by w0m on Thursday July 30, 2026 @05:36PM (#49116148)
> Why do you feel that creating the LLM from scratch is sufficient to trust it

If you're doing the training yourself, you at least have a verifiable supply chain and an audit trail. Today, we have no idea if a black-box model handed to us is coded to recognize specific domains or patterns and back door an application in a sneakily targeted way.

Black box is a black box. Many open-weight models clearly haven't been trained or created in the manner their creators claim---which raises the obvious question: if they lied about the recipe, what else did they lie about? Putting those models in a production capacity scares the living crap out of me.

That said, building from scratch isn't about achieving mathematical perfection or manually auditing 15 trillion tokens---that's impossible. It's about eliminating third-party supply chain risk and having actual governance over the pipeline.

Of course, that doesn't mean we can magically guarantee gradient descent won't produce weird emergent behaviors, or that we can blindly trust OpenAI not to backdoor things. But at least with the latter, you're making a calculated operational decision rather than blindly trusting an opaque black box of entirely unknown provenance.

Re:Kimi-K3 Technical Report [pdf] by JacobAsmuth on Sunday August 02, 2026 @05:23PM (#49148482)
Why do you think you've prevented third party supply train risk when you're training on a 30+ Terabyte blob that you have little understanding of its contents. It's essentially entirely composed of unfiltered user input.
Re:Kimi-K3 Technical Report [pdf] by mdp2021 on Monday July 27, 2026 @03:59PM (#49074832)
> What's to prevent the LLM from

A NN per se is a file... The executable that runs it can "act"...

Re:Kimi-K3 Technical Report [pdf] by shrubble on Monday July 27, 2026 @03:25PM (#49074461)
As mentioned, it is a "large enough company" already; they have full time sysadmins running things.

Adding another rack beside the VMWare cluster, managing any storage/networking issues, etc. will be incremental costs; they already have a pager (probably not a pager not anymore just an app on their phone) like rotation schedule etc.

Re:Kimi-K3 Technical Report [pdf] by russell_h on Monday July 27, 2026 @12:55PM (#49072404)
> However, I don’t trust hosted LLMs for anything that needs to be private.

Why not? Do you trust AWS with things that need to be private?

Re:Kimi-K3 Technical Report [pdf] by Kevcmk on Monday July 27, 2026 @01:01PM (#49072494)
More than I trust frontier labs. AWS doesn't need to recoup 9 digits USD of capex
Re:Kimi-K3 Technical Report [pdf] by senderista on Monday July 27, 2026 @01:08PM (#49072590)
So you can just use Bedrock?
Re:Kimi-K3 Technical Report [pdf] by retinaros on Tuesday July 28, 2026 @10:15AM (#49084232)
aws its their business to make ur data safe. frontier lab business is to use your data and mine to train models
Re:Kimi-K3 Technical Report [pdf] by GodelNumbering on Monday July 27, 2026 @12:59PM (#49072456)
> And hire 2 or 3 dev ops to keep it running

Not a devops but I'd say one full time is already too many.

Re:Kimi-K3 Technical Report [pdf] by dboreham on Monday July 27, 2026 @01:10PM (#49072619)
Yes but zero is not enough and where do you get a fraction of a competent dev op from?
Re:Kimi-K3 Technical Report [pdf] by layer8 on Monday July 27, 2026 @01:15PM (#49072700)
From the other dev-op work you’re doing.
Re:Kimi-K3 Technical Report [pdf] by JacobAsmuth on Monday July 27, 2026 @04:02PM (#49074850)
Now you're thinking like management!
Re:Kimi-K3 Technical Report [pdf] by ayewo on Monday July 27, 2026 @01:50PM (#49073186)
Understood but sharing your existing devops resources with this will soon become a bottleneck especially when any major downtime will keep several engineers (and long-running agents) blocked from any meaningful work until availability improves.
Re:Kimi-K3 Technical Report [pdf] by layer8 on Monday July 27, 2026 @02:08PM (#49073411)
Not my experience, from an SMB that maintains its own hardware and services. You have a certain contingent of competent engineers who distribute their work across projects, and it generally works out fine. Or course you plan with some redundancy and fall-back plans in your systems.
Re:Kimi-K3 Technical Report [pdf] by ayewo on Monday July 27, 2026 @04:11PM (#49074940)
The top poster mentioned LLM spend of millions/month to justify the estimated capex of $6m to self-host Kimi on own infra.

Add to this number another $1.5m/yr in opex, so not sure I’d call such an enterprise wealthy enough to spend those kinds of sums on LLMs an “SMB”.

Re:Kimi-K3 Technical Report [pdf] by twobitshifter on Monday July 27, 2026 @07:28PM (#49077005)
colocate and split the cost with others.
Re:Kimi-K3 Technical Report [pdf] by lumost on Monday July 27, 2026 @01:00PM (#49072483)
There will be cloud/SaaS vendors who have lower cost of labor/capital due to automation and financing terms.

Having these models in the open caps the inference margin.

Re:Kimi-K3 Technical Report [pdf] by slicktux on Monday July 27, 2026 @01:07PM (#49072575)
Just like that new jobs created by AI! Localized model maintainer/technician.
Re:Kimi-K3 Technical Report [pdf] by toomuchtodo on Monday July 27, 2026 @01:07PM (#49072585)
You’ll slap some training on existing technologists/infra/sysadmin folks and perhaps have a support contract for the edge cases (hardware troubleshooting and advanced replacement).

(managed an entire data center building with thousands of servers a lifetime ago with ~2-3 other people, it’s only gotten easier over the last two decades imho)

Re:Kimi-K3 Technical Report [pdf] by clint on Monday July 27, 2026 @12:55PM (#49072403)
Just let it manage itself, what could go wrong! :)
Re:Kimi-K3 Technical Report [pdf] by JimmaDaRustla on Monday July 27, 2026 @04:33PM (#49075156)
Why would a singe system require 3 full-time dev ops?
Re:Kimi-K3 Technical Report [pdf] by kcb on Monday July 27, 2026 @02:20PM (#49073575)
You also need a place to put it. With liquid cooling and extremely dense power capability. Your typical colo or closet server room isn't going to cut it.
Re:Kimi-K3 Technical Report [pdf] by reckless on Monday July 27, 2026 @12:52PM (#49072364)
I think the licensing that would likely apply to a company that's able to afford ~$6M rack and the associated infrastructure muddies this somewhat
Re:Kimi-K3 Technical Report [pdf] by petu on Monday July 27, 2026 @01:03PM (#49072535)
I think internal use is allowed at any scale in the license?

> 4. The requirements set forth in Sections 2 and 3 do not apply to: (a) internal use of the Software, defined as any use that does not make the Software, its outputs, or its underlying capabilities available to third parties; [...]

Re:Kimi-K3 Technical Report [pdf] by jsnell on Monday July 27, 2026 @01:31PM (#49072928)
It's very hard to make sure none of the outputs are ever made available to third parties. Source code can end up widely distributed (e.g. client-side js, open source). Prose will frequently get shared across organization boundaries (e.g. emails, websites, documents).
Re:Kimi-K3 Technical Report [pdf] by stymaar on Monday July 27, 2026 @03:06PM (#49074234)
License for LLMs already have very little strength (it's not clear at all that they have any legal basis whatsoever) an AI lab is very, very unlikely to sue you for violating this kind of license clause for the reason you mentioned. It's clearly worded in a way to deter people from running a third party inference business out of it, they would have been much more explicit if they wanted to deter people from running it for internal use.
Re:Kimi-K3 Technical Report [pdf] by vidarh on Monday July 27, 2026 @01:00PM (#49072480)
As far as I can tell, license fees are only applicable if you have more than 20m USD/year revenue from services provided using the model, or serve more than 100m users.
Re:Kimi-K3 Technical Report [pdf] by jdsully on Monday July 27, 2026 @06:57PM (#49076636)
Its an open question if model weights are even copyrightable. So the license may not mean anything.
Re:Kimi-K3 Technical Report [pdf] by baron3dl on Tuesday July 28, 2026 @12:48AM (#49079473)
I was wondering about this myself, specifically where does it make sense, so I spent a few hours working through it with Claude and wrapped it up into a writeup and interactive model. https://3dl.dev/kimi-k3.html