The moat is GPUs/TPUs, ram and compute. When you need $200k of compute to run the frontier models, you are likely to be renting access to generate your tokens.

Probably close to $1M+ if frontier models are truly 5T and above params.

Current RAM is no moat. Upcoming Compute-in-RAM is. And nobody has that built up at scale yet. So, while a moat might become possible, we only have yesteryear's hardware as "moat".

Can you explain what compute in RAM means?

I thought HBMs are technically the fastest possible thing for inference.

The academic term is now "Compute-in-Memory". That's MOSTLY what has suddenly been hiring SRAM and DRAM designers across the industry this year. It is part of the fast evolving field of doing inference efficiently. This has been all the rage in several Electronics conferences this year.

Please don't rely on my [inferior] explanation and search for yourself (start here for insance: https://www.google.com/search?q=what+is+compute+in+memory+an...). Asking about a new field is much better done with AI these days, because you can get it to explain just the interests and questions you personally have, rather than a generic and boring explanation.

In short, better than HBM is not having to move data in and out of memory at all or at least not as much. The compute part in inference is much simpler than the data hauling part.

Couple of examples: Check out Axelera's Europa board (upcoming) or their Metis board (existing). Or check out Mythic.ai which FINALLY managed to raise funds in Nov / Dec 2025. Ironically, her PhD thesis had been gathering dust before that.

That is not a moat. (and it is not theirs either) It just means that their models are highly inefficient that it requires a lot of compute to run them.

Open weight models such as Deepseek, GLM are just as good as the frontier closed models and already we have bank such as Revolut migrating off of frontier closed models and self-hosting. [0]

[0] https://x.com/guilleflorvs/status/2117379073785696395