AND developer cloud has an image for K3 running it on a single MI300x. Offloading to system ram can be quite decent.

I cant say how well that image works as I'm still waiting for them to approve my gpu access

Do you need 450k context?

I usually keep under 200k if I can help it for Claude/ open ai

And lower locally on my Mac to save on prefill.

No, also because that would mean I have to read up on rope scaling or whatever to go beyond the default 256k context, which so far I’ve avoided. But you can allocate it as 220k context for 2 parallel Pi workers, that really speeds things up! Or as ~8x 50k context for dedicated parallel document processing.

Possible data center bans incoming + tons of GPUs sitting in warehouses due to lack of power infrastructure to power them.

Perhaps a flood of 8x B300 will be homelab ready in a few months

No, they'll be shipped off to Asia where technology is embraced.

The rise of China due to hand-over-fist ceeding of power and influence fueled in equal parts by the right and left is so astonishing it'll certainly go down in the history books (written in Mandarin, of course).

Bonus points when the morons are doing so because they get all their political news from a Chinese platform. The [Americans] have gone insane.

The rise of China is due to the tyranny of the markets and the fact that firms laser-focused on quarterly growth buy and liquidate those trying to do responsible, sustained growth. Those that sold their factories were able to swallow domestic competitors who didn’t do so.

The political system just does whatever the captains of industry tell them to do.

> a flood of 8x B300 will be homelab ready in a few months

Haha, that's beyond wishful thinking, I'm afraid. We're not even seeing a flood of A100s (6 yo and several generations old). Every GPU is still being squeezed right now, and the trend will probably continue for the foreseeable future.

Last I checked AWS still sells EC2 instances using 2018 era GPUs. Most people underestimate just how much long tail demand there is and how long these cards last, despite the higher power consumption.

Yes, but if data center space gets tight, it creates an incentive to replace old inefficient models with better ones, i.e. sqeueeze more performance out of limited data center space.

IIRC some companies were running campaigns like "we buy your old hardware so you won't sell it in aftermarket and crash the prices". It all ends in landfill.

I wouldn't be surprised if RAM cartel or nvidia will attempt something like that, plus nvidia already talks about license-locking GPUs.

Also, again, IIRC, IBM hardware already works this way. There are almost no aftermarket when I last checked for them because it's all locked up and useless.

Maybe, but A100s and H100s were air cooled. Space is just part of the problem.

The world is bigger than the US. If the political climate in the US obstructs the build out, the build out will occur somewhere else.

>The world is bigger than the US

This is cope. It's cute that you think it's true, but it's not really much bigger than the US.

It’s called pacing the frontier

I agree, but it would have to be a place with plenty of energy and the only such place outside the US is a war zone.

Well, lots of countries have energy surpluses from hydro or cheap coal.

It would certainly fit the pattern of Bitcoin farms clustering near dams.

[delayed]

>Possible data center bans incoming

A lot of the data center build out has been bipartisan. Unless Dems realize they need to listen to their constituents to avoid a 2028 bloodbath (they probably won't), I wouldn't count on that to save us from RAM/GPU prices.

There are 300 racks of nVidia A100 servers for sale from a refurber right now on eBay...

You could easily run it with SSD streaming on as little as 192GB RAM (The dense portion is around 120GB or so at native precision, and the remainder would fit KV and a cache for the most commonly referenced experts.) This fits Gorgon Halo, which is a reasonably priced platform. It would be slow, but still usable for high-value tasks.

Hmmm. I instantly discounted this for my 2x DGX Spark cluster, but you're right. In 256Gb I could probably have large context and stream the experts from disk. I imagine it would get me like 10-15tps with a 30 minute prefill though.

Excuse me, could you elaborate how you came up with 192GB of RAM? I almost thought they downsized model, but it's still 2.8T tokens. Are we talking about not-yet-out 1-bit quant or assumption that something like ds4 (daskstar, not deepseek) will heavily optimize it?

Not a trolling question, genuine interest.

it's a mixture of experts so it's somewhat viable to stream weights from nvme

It's not loading the full 1.5TB model into VRAM or system RAM, it's streaming parameters on demand from SSD. This is worthwhile since the MoE portion of the model is quite sparsely accessed. It's been demoed on a 128GB M5 MacBook Pro already with lousy performance (0.3 tok/s or so) because it had to load at least ~25GB from SSD per token (comprising the entirety of the sparsely routed active params) and potentially even reload some of the dense weights, depending on what exactly managed to fit on that platform. With 192GB you'd avoid any reloading for the dense weights and even cache a few layer-experts.

Wow, TIL. Is there any calculator or rule of thumb you use to figure out the expected performance of a model on some DRAM/SSD split? For instance I am wondering what the performance could be like on a 512GB M5 Mac Studio.

The rule of thumb is to just test it on your chosen platform, with your workload. SSD performance matters a lot obviously, you'll want to run PCIe 5 if at all feasible, or a recent M5 Pro/Max/M6 Mac with high storage capacity (lower tiers have much worse performance). You can figure out a worst case easily (assuming that the fraction of sparse params you manage to cache in VRAM or system RAM has a fair chance of being picked among the total) but in practice you do quite a bit better than that - which is why even very small caches can be potentially worthwhile.

Yes, I understand the principle, but will it be usable? E.g. in my personal opinion border of usefulness starts somewhere at 7-10 tok/s. I currently have 192GB and 24GB VRAM, and I can run Qwen-Flash-Next (4 bit) and GLM5.3-Flash (4 bit) at 25 tok/s and 7 tok/s respectfully (with quantized cache on beellama). Parallel queries worsen result significantly but can push aggregate to around 50 tok/s.

I can't, however, run DS4.1-1017, for example, but maybe I didn't find proper options and quantization yet.

So I'm really interested how to be sure what can/cannot fit in specific hardware (for example, I want to buy a Mac Studio with 64 or 128 GB RAM, what MoE model it would run at comfortable speed? Will it be qwen-flash-next or something bigger? And so on). So far websites with "benchmarks" were really terrible in that regard (and huggingface calculator even worse!)

> in my personal opinion border of usefulness starts somewhere at 7-10 tok/s.

This is expected if you're actively steering the agent, but what about running unattended tasks - perhaps multiple such tasks in parallel, such that you can stay engaged by "attending" to each of them in turn? Then you're not restricted to such a requirement, and can benefit from much slower (individually, not wrt. total throughput) inference streams.

> I currently have 192GB and 24GB VRAM

I'm not familiar with beellama, but this seems like it could be the perfect setup for Strata which people are saying good things about, including for running DS4.1F (at least quantized). The whole point of it is to effectively leverage heterogenous RAM/VRAM setups. Of course llama.cpp itself is doing a good job of trying to catch up, but they've yet to support efficient SSD streaming (mmap doesn't quite cut it) in the main branch.

I run this exact setup with 24GB vram and 288gb ran. I get 25 to 28tps. The catches you have to run the 2.25-bit version currently. Strata is not optimized to run the 3 ir 4-but versions, at least not yet. With a 256K context window and vision I think it uses around 135 MB— I can’t remember if that’s total or just ram.

> I want to buy a Mac Studio with 64 or 128 GB RAM, what MoE model it would run at comfortable speed? Will it be qwen-flash-next or something bigger?

This depends on how you plan to use the machine... but Qwen-flash-next is right at the top of what a 128GB Apple Silicon can do.

QFN with 4bit quantization will run "comfortably" with all weights in memory and a 128k context on a 128GB machine. On M5 Max w/ 128GB, that will give about 75 tok/sec (using oMLX with MTP enabled). It spits out text much faster than I can read.

However, this truly uses all the RAM. You'll have like 8-12GB left for the OS and apps, which isn't really enough to actually use the machine as a workstation while the model is loaded. But it would work fine as an inference server with Opencode running on your laptop. (Hence, "how you plan to use it?")

Alternatively, you can offload QFN's n-gram table to SSD (this is just a settings option in oMLX), which drops generation speed on the same machine to ~60-65 tok/sec (assuming the model lives on the internal SSD) and reduces memory usage to ~90GB with even a large context. That leaves plenty of RAM to use the Studio as your main workstation. Also, you may run much longer contexts because of the extra memory headroom.

Second alternatively, you can use oMLX's expert offloading feature to keep 50% of experts in memory (with n-gram offloading also). This reduces the RAM footprint to a little over 32GB + context. Offloading the experts drops generation speed to around 20-25 tok/sec on the same machine as above, but it would allow you to comfortably run on a 64GB machine, while still using it as a workstation.

I have tested the 50% expert offload case on an M1 Max / 64GB, and the inference speed is only a little slower (more like 18-20 tok/sec) than M5 Max / 128GB because in both cases the bottleneck is SSD read bandwidth and the GPU is spending a lot of time idle.

One last thing: GLM 5.3-flash doesn't use an n-gram table, so you can't use that trick to reduce memory footprint. You'd need to use expert offloading to make it fit a 128GB machine. Since it has over twice as many active parameters as QFN (which doubles the required SSD read bandwidth), my guess is that a 128GB Studio would not be much faster than your current setup for that model.

All these very large models are sparse mixtures of experts (SMoE), where typically only 2-3% of weights are active for a given forward pass.

Usually there are some static experts in each attention layer, plus a whole bunch of dynamically-routed experts. Typically some form of LRU cache is used for the dynamic experts.

So you can keep only the static expert(s) and routing layers in memory and then stream the dynamic experts from SSD, at a modest inference speed cost driven by the latency required to load layers from SSD when they aren’t already in the memory cache.

The LRU cache works reasonably well because a given task will often hit the same experts on each iteration of the forward pass.

For example, on a Mac Studio, Qwen3.8-flash-next (180b total parameters) with 50% of experts in memory and n-gram offloading has about 1/4 to 1/3 the inference speed of everything in memory. That requires 32GB of vRAM for the weights (plus a little extra for context, obviously)

Models this size aren't targeting hobbyists running models in their bedrooms on gaming/surplus hardware bought with disposable income.

This model is for researchers who already have racks of equipment, inference providers who sell access to models, and for businesses who must (or choose to) use on-prem models. All of which have to pay to use the model if their revenue rises beyond certain thresholds: https://huggingface.co/moonshotai/Kimi-K3.1-Pro/blob/main/LI...

It's only "model as a service" providers beyond the stated threshold that have to pay. No issues for any kind of purely internal use.

> more than $20 million…in monthly revenue

Sign me up!

The model-as-a-service clause (which is new to the K3 series) actually only requires $20 million over 12 months - i.e. yearly revenue.

abd why should we expect this to be the case? I can atleast buy old server hardware for cheap. GPUs are currently locked down the whole industry.

Considering the likelihood of Moonshot being able to collect a judgement for violating these terms in an American court is basically zero, and the relative ease of finding the weights without the licensing agreement, "have to pay" seems very generous.

Most enterprises capable of running that much gear won't take that risk even if it is unenforceable. Oracle has made a great living threatening audits of companies who download its updates.

Yeah, but only Western companies who might have to defend themselves in court. I have no doubt that pirated copies of Oracle DB and the JDK are frequently used in China.

That defense won't work in court.

> Considering the likelihood of Moonshot being able to collect a judgement for violating these terms in an American court is basically zero

Why? Americans are always yapping how China growth is basically fueled by IP theft, are you saying that it works both ways?

Or, you can just not pay and save that money to defend yourself in case they do sue you. Because it isn't established that model weights are copyrightable.

It's a license agreement, not copyright.

From your link I do not see anything about paying from the model, only displaying a model logo in some (but not all) high user high revenue circumstances. For example, the revenue threshold is $20mm/month to display the logo, but only if the external users use the model. If I make an iOS app that earns $20/mm mo and has more than 100m users with this model, do not display the logo, unless the model itself is available to those users.

The benefit is that you can use these models on a variety of different providers. You're not stuck with a single proprietary provider. Aggregators like OpenCode Zen and OpenCode Go can shop around providers on these. OpenRouter shows you the costs of a bunch of the providers, and you can route to the cheapest ones.

I regularly use together.ai with models like this, where they're fully self hosting them and I can turn off all data retention, at a fraction of the cost of proprietary models. I recently discovered abliterated models (models edited to not refuse to work on security stuff) and hosts for them like refuseless.com, and that's an option for open source models, not for proprietary ones.

> Aggregators like OpenCode Zen and OpenCode Go can shop around providers on these. OpenRouter shows you the costs of a bunch of the providers, and you can route to the cheapest ones.

Unfortunately due to their licensing requirements for modal-as-a-service providers, the pricing can’t go lower than Moonshot’s from what I can tell.

This is still great if like your example you trust together.ai more, or if different providers have better speeds/cache rates (though usually the original lab has the best cache hit rate).

And obviously if you’re a large enough service to afford to self host or finetune it then this is good.

Maybe for the API rate, but for OpenCode Go they're regularly making deals to get users a lot more than their money's worth on open source models.

Sometimes it can be so hard to tell and track because of the cache differences.

I didn't dig into Kimi's MaaS legalese, but i dont really understand how that knee-high fence is supposed to work long term.

All the service provider needs to do is provide the option to rent out VM's with access to the inference cards.

Have the customer provide the model.

Maybe provide a bit of a speedup, by letting the customer mount the model by hash.

Sure, that all sounded way too technical 10 years ago, and somebody would have said that's already too much annoyance to deal with for average customers.

Reluctantly i'd agree.

That was 10 years ago. Nowadays, with some proper docs, this entire set up can be up and running in 5 minutes.

There's no need to use ablated models right now for cyber work. Several open weight models don't refuse cyber requests (DeepSeek, GLM, Kimi K3, etc). And as you said, OpenRouter has many 3rd party ZDR inference providers and several of those serve these models.

Ablated models are not any more inherently knowledgeable about "forbidden" topics, either. Since such topics are not generally trained for, you just get weird creative hallucinations.

The problem with looking at different prices from different model providers is a lot of the cost is generally driven by cache hit rates. That’s where you can accumulate expense and the cache hit rate can very wildly from one provider to the next.

It’d be interesting to think if device swarm serving of these models ever becomes possible - are a few projects that seem to be just starting on this.

It seems more plausible to me than wherever the frontier is at any given time getting small enough to serve on one local card directly

[delayed]

We're running Kimi 2.8 on a 4x AMD MI300A single server on-premise that cost about $110k.

We will break even from our old Claude spend in about 2 weeks.

It services 30-40 analysts and staff concurrently and is faster than any of the SaaS out there for us.

Wow... I'd like to know more. 40 analysts not spending 100USD/month is 4k/month saved. That means you break even in 27 months. But you probably used Claude much more than that.

Also what harness do you use? Any problems you encountered on the way?

Oh boy... we are talking about insane token spends in corps at the moment. 100$ is a blip.

I think you should compare with api cost? I am using $200 plan and it is probably around $12k in token cost....

Yip... I thought these numbers were absurdly exaggerated since I'm on a sub plan, until Anthropic gave the $50 credit back in the Fable days. I burnt through that in a couple of hours.

It's one of the many reasons this is all obviously just a bubble. Cloud gouges people, but it's a relatively small gouge that doesn't scale. API token costs are just LOL and an LOL that scales up with everything. As soon as the novelty of LLMs starts to fade and they become just an integrated part of society, there's going to be a rug pull, but in reverse.

Just one of our analysts was burning around 30-40 million tokens per project.

EDIT: per project, not per month

EDIT 2: digging out an old bill from 8 months ago when we bought the server, $14k per month is what we were running and it was going up, a lot.

Anthropic is very good at either convincing or forcing (I’m not sure which) companies to pay for tokens rather than for subscriptions. My old job moved from subs to tokens and everyone started running out of budget mid month. It was a bad move for a company that laid off all of QA and 2/3 of Engineering and committed to everyone making up for it with LLMs.

I don't get this, why do people pay for API tokens at all? What is the reasoning?

Zero Data Retention is the big one I think (and also the ToS don't allow multiple personal accounts for businesses anyways)

They don't have the choice to begin with

I don't think Anthropic or OpenAI offer subscriptions to enterprises - it's a hobbyist/solo-entrepreneur kind of thing. I seem to recall SpaceXAI had some kind of hack going where they were trying to use subs with Anthropic and then got cut off.

Enterprises also need very stringent IP protections that you don't get through the subscriptions. Even the Anthropic-API per token license isn't great - sometimes you need to go to 3rd parties like AWS Bedrock to get guarantees regarding no multi-tenant GPUs.

You can have subscriptions up to 150 seats with Anthropic.

You are forbidden from the subscriptions on enterprise

Claude Teams exists.

No Zero Data Retention there.

I have only ever paid for tokens and never used subscriptions because I need to know that I can generate sufficient real world value at the true cost of inference.

How bad did the electric bill get hit?

We have a data center already and we're in heavy industrial manufacturing. It didn't even register in our usage.

Are they happy with Kimi 2.8's reasoning ability? I haven't tried it.

It's worked out. Definitely not frontier but it's the largest model that works reliably on our hardware. We also try some smaller ones when they come out.

Kimi is also a lot more willing to do whatever tf you ask it such as vulnerability scanning or PoC development. Or reverse engineering.

Plus on prem can go find abliterated models.

It's an advantage that keeps me coming back for local models when I can't get around Claude or Codex's little blockers.

Very cool

There's no Kimi 2.8 as far as open weight is concerned. WTF. This must be a bot and other bots.

Maybe they're talking about parameter count?

If you take the kimi plan you would have seen kimi 2.8 preview alongside kimi 2.7

you are aware that hardware gets better and cheaper, right ?

2 gb300 node actually. Or 8 gb300 cards since 1 node has 4.

> Black Friday is coming soon though

A few weeks before Black Friday a couple years ago, I noticed a listing on Amazon for a high-end AI card for around $40,000 (maybe a B100, don't recall). The listing had the automatically inserted Xmas season note "Free returns until Jan 31." I spent a serious minute pondering if I could float the cash for the ~40 days between the credit card bill being due and returning the card. :-)

I have managed to boot up these models for 1-2 users on just a couple B300s. Which costs a lot still but it's not that bad.

Depends on number of users - for a single user with a couple of sessions, you don't quite need that much. Bit of a waste of hardware to only serve this for single users though.

It's not an open weight model (check the license, and the definition of that term if you disagree).

[delayed]

I assume you mean condition 2 - where they want anyone serving it as a service (to others) and earning more than $20mUSD per year in revenue to contact moonshot and enter a separate commercial agreement.

wish they just said "we are better than openai and anthropic, fuck you sam and you dario"

"Pacing the liquidity frontier" :D brilliant. Will be using that.