chrisringrose

Quick mission statement, since it explains everything below: Deltafin runs the full, never-pruned, 2.8-trillion-parameter Kimi K3... all 16 experts, every token, no shortcuts, nothing re-encoded down to "close enough." Speed is not allowed to come from quality.

A few similar K3 projects have shown up claiming to be the same thing, just faster. But they all get there by shrinking the model - quantizing it down, trading fidelity for speed, calling the results close enough. Those are great projects, and have real value - but they're very different projects. Deltafin's purpose remains the real full K3, unaltered and uncut, as fast* as possible.

  * it's not fast

Since this was posted, throughput on the M1 Max laptop went from 0.0141 tok/s to 0.2847 tok/s - about 20x, and still climbing. Most of that came from routine hardening: fixing a storage race, a rust/c++ rewrite, and tolerating a float32 rounding drift that looked like a routing bug until it wasn't.

None of that is a moonshot (get it?). It's the boring, necessary work of making something this huge survive on consumer hardware.

Now one huge thing to get out of the way, because this is about half the comments: is this a viable business product, the kind of thing that makes people in suits happy because it helps them sell widgets? No. My interest in computing goes beyond that, and I suspect a lot of yours does too. I'd like to think that's still what hacker news is largely about.

A research project usually isn't the useful thing itself. It's what falls out of chasing it.

It is genuinely hard to squeeze even a small speed gain out of a model this size on a machine this small. Yes, an M1 Max is a good laptop. But Kimi's own recommended setup is 16 nodes at ~300GB of VRAM each. That's $2,000,000 of hardware. Next to that, this laptop is an ant lifting a car. And that is the point of this project - the insane challenge.

My hope is that if we push as hard as we can to get the largest public model ever released running on the smallest machine that can plausibly hold it, some of what falls out along the way could be useful elsewhere... maybe to smaller models, on smaller machines, running faster than they do today.

But first I wanted to find out whether this was even possible at all. K3 is the largest public model that exists. Nobody had run something this size on a laptop before. The closest comparable projects like colibri and DwarfStar ran quantized models at half this size. Turns out: yes, it's doable, but slow af.

But I've spent a long time frustrated with the quality ceiling on local models, and quantization is a big part of that ceiling. The hallucinations. The reasoning that almost holds together, but then doesn't.

So I'm done compromising on that axis. This isn't going to be another quantized, slightly-worse copy of the real model wearing its name.

Maybe no techniques we’re using are groundbreaking at all in the end. Bummer. But, at least I tried. Try and fail at life, people. That’s okay. Just try, and at least you’ll have lived a little. It's a decent way to spend the time you get.

If people want to keep following this, I'll post updates here instead of just on Reddit like I have been.

We've made some updates to deltafin since this was shared 5 days ago. I wonder if you'd be able to get better speeds now? https://github.com/gavamedia/deltafin

WASTE reads about 17 GB/token versus Deltafin’s 25.8 GB/pass—roughly. That's 34% less expert traffic, and some could argue a 34% reduction in quality.

I've had legit discussions here about real tech topics (for example web apps vs standard EXE apps), and the guy who continually disagreed with me clearly down-voted every comment I made, and replied with out-dated counterarguments. I wound up losing almost all my karma from one discussion, all while remaining civil.

It's very discouraging to be new here, and try to openly express an opinion. It should take more than one to down-vote a comment successfully.

This is a win for web apps. This is proof that traditional desktop apps are dying.

- I'm not sure why you say that. I do online banking all the time. It is possible to code something secure and tight, but I'll admit it's not easy. I think security will always be an issue, even with native apps.

Local Data - Storage is limited & it can be manipulated

- True, storage needs to be unlimited. As for manipulating it, this is also true of native apps. Without jailbreaking, I can manipulate a save file for almost any iPhone game. The security issue (again) depends on the developer, and how they protect the data.

Syncing offline apps - Determining which is the latest version etc

- Yup, that's a challenge too. But also a challenge native apps face. When I make a note on an offline iPhone in the notes app, and then edit the same note on my Mac, it creates a duplicate when the iPhone is back online. It should probably be up to the developer how to handle this, whether the app is native or HTML5.

If the browser vendors won't implement something, it won't get implemented. This has also lead to

- Yes, this I think is one of the biggest challenges. People buy a computer, it comes with a browser default, and they never change or update it. It's hard enough to get the W3C to add new features, let alone browsers to implement them, and even harder still to get users to update their browsers.

Format Incompatibilities e.g. try audio/video tags across the major browsers

- Yup, similar issue.

unhealthy competition between browser vendors, example: WebGL@InternetExplorer - Agreed. Especially when one browser stays from what's supposed to be "standard", yet rarely is.

its limited to 3 languages: HTML, CSS, JS (and its derivatives) - Agreed. Google's working on a compiled web language, but it will only work in chrome. They're attempting to make it a standard, but I'm not sure if any other browsers would put the effort into supporting it. And a whole new system like this would take years to implement. But still possible....

In other words, the number one selling laptop today can't run any desktop applications - just web apps. I'm a developer, and I simply can't ignore this.

It's time to focus our efforts on fixing the obstacles web apps face, instead of doubling-down on desktop/native apps because they're easier to make, or run slightly faster.

I saw your previous comment earlier, and even gave it an up-vote. I'm new to Hacker News, but perhaps some day I'll have the right to down-vote; and when I do, I won't use it simply because I disagree with someone.

http://www.w3schools.com/tags/tag_video.asp

Don't bother making anything - there's no chance it will be successful, no matter what. Life sucks, then you die.