Re:Open Source AI Is the Path Forward
by verdverm on Tuesday July 23, 2024 @10:55PM
(#41053235)
Analogies are always going to fall short. With LLM weights, you can modify them (quant, fine-tuning) to get something different, which is not something you do with compiled binaries. There are ample areas for collaboration even without being able to reproduce from scratch, which takes $X Millions of dollars, also something that a typical binary does not have as a feature.
[ Parent
| on HN ]
You can absolutely modify compiled binaries to get something different. That's how lots of video game modding and ROM hacks work.
And we would absolutely do it more often if compiling would cost as much as training of an LLM costs now.
I considered adding "normally" to the binary modifications expecting a response like this. The concepts are still worlds apart
Weights aren't really a binary in the same sense that a compiler produces, they lack instructions and are more just a bunch of floating point values. Nor can you run model weights without separate code to interpret them correctly. In this sense, they are more like a JPEG or 3d model
JPEGs and 3D models are also executable binaries. They, like model weights, contain domain specific instructions that execute in a domain specific and turing incomplete environment. The model weights are the instructions, and those instructions are interpreted by the inference harness to produce outputs.