mrashad74

Imagine your printer has it's driver failing every 2 days but thankfully it can generate YouTube videos

Typos are a sign of human intelligence :D

this would be a nice challange but from what I did, very doable.. not a usable speed maybe.

looks nice! what languages does it support?

100% it's on my queue of devices

happy to hear that, feel free to contribute.

not usable ones tho, but something like Karapathy's LLM may run

a sentient tamagotchi is not far from us..

Limits pushed again this time on phones..

Qwen3.6-35B-A3B at 4.9-5.7 tok/s Qwen3-Next-80B-A3B at 2.8 tok/s

And targeting 10 tok/s for both, 35B modified is on hugging face and 80B is uploading..

Neat, I like it, do you have github repo for it?

I had this case in fine tuned small models (up to 27B) in finance and legal, over bigger models (cost-wise, on-prem for a regulated institution, and better perofrmance because we owned the data pipeline)

Decent!! Also I love the idea of the live chat, would you mind if I put a reference to it on my repo?

actually the pi zero is a nice target to play with next, will give a shot and report

I'm also working on this thread in parallel because it has interesting business aspects, would be happy to see your progress when you release it

I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.

a cluster of vapes running an LLM would be insane. a single vape of these though may run a small LLM like few hundred thousand parameters, worth tinkering with.

it was a fun name honestly, I see people using the term in different funny contexts on twitter/x so I didn't think much actually. but good point.

ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.

I agree, this is why in the further experiments I was putting LFM as the main model not qwen, qwen was nice to prove the point for me, but as I wanted to do more work it became unusable and bad quality, LFM is fine, Gemma didn't perform well but I didn't go far with it, I need to check the others,and I tried MiniCPM but not on this project.

Thanks for the pointer!

running your own hardware and extracting value out of every piece of hardware is important - one for efficieny, not every application needs smth big - and two for applications that need privacy and similar controls

let's do it, I'm on twitter @rashaddism

if you're having some cards already try to explore what people are doing with weight streaming and model compression, I got good results on older cards with far better models than the 16B

and the new thing will be "will it run qwen" :D

streaming weights of the CD drive should be top 10 blessed engineering hacks I've seen in the past 2 weeks.

long live TI-84 Plus, the first sentient calculator

yes I worked with this already in legal, banking and industrial automation. small models with a specified task work well on modest hardware.

Check LFM as a starting pointer : https://www.liquid.ai/models

Agreed, and it's a new use case to me because actually in the cases I had the clients were okay with getting GPUs and handling the data mess for us, but none of them requested fine tuning, I believe trying to fine tune will be insanely complex from a regulatory prepsective.

actually your methodology inspired me to try smth related to small models and harnesses I've been thinking about - and seems you got the main pieces (the idea was trying to see what custom harnesses can extract more accurate reasoning from small models)

this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.

I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot

the kind of one shots you get with fable 5.5 :D (it was a double typo)