Farmadupe2

For the last five months or so, opus and fable have both been able to do this task as a soneshot prompt. An inference engine is almost pure compute and to a first approximation the only operation your hardware needs to support to do llm inference is multiplication [the rest of an inference engine is "some trig"].

So to get any model working on $OLD_HARDWARE, the hardest part of the task is now sourcing the hardware itself and giving the agent ssh access -- a modern agent will be able to source a compiler, write the implementation, optimize it, test in qemu (or any other simulator) and push it onto the device.

Example prompt: "write an inference engine for qwen3 0.6b in pure assembly that runs on this device. Write a script to attach the weights to the resulting binary. Prefix is supplied as a stream of u32 token IDs on stdin. The engine streams u32 token IDs on stdout."

Yes if parakeet would work then thats the answer and the code has already been written.

Otherwise the answer is really mechanical. It's will run as fast as you can get the weights into your cou's ALUs. since it's 30b-a4b, if you have 30G spare ram you can load the whole thing and it will run about as "fast" as any 4b model (qwen3.5-4b, gemma4 e4b etc).

Otherwise you will be streaming from disk/SSD which may or may not be very slow.

Almost all ML models come with reference python code that will run on most architectures.

If you want an implementation optimized for a certain architecture the cost is between 10-30USD of opus tokens and is oneshottable: ("implement Nvidia nemotron 3.5 transcribe for the intel npu on this machine. Use the hf transformers implementation as reference. The finished artifact takes pcm on stdin and utf8 chunks on stdout".)

You can substitute your old intel CPU by modifying the prompt by replacing "intel npu" with "the target cou on this machine ... Your task is complete when the model decode performance is 80% of the memory read speed of this machine"