> I want a more challenging device, what do you think?

One of those vapes with a screen on it.

challenge accepted.

I have about 12 of these Rifbar Mixpads saved up: https://rifbar.com/product-mixpro

They have a color LCD display that looks unnecessarily good considering what it does. You can see an example of one of the animations at the 52s mark in this video https://www.youtube.com/watch?v=peNyiLt5SKk&t=52s

I've got no idea what's inside of it yet but I'm open to tinkering if someone can walk me through it

I doubt any vape is powerful enough to run any LLM on its own, but idk if it's possible to run one on a cluster of vapes.

a cluster of vapes running an LLM would be insane. a single vape of these though may run a small LLM like few hundred thousand parameters, worth tinkering with.

Vapes typically have less than 100KB of RAM, though I haven't been able to find specs for the Rifbars yet. But I assumed even if they're 1MB that's still not enough to run any kind of LLM right?

not usable ones tho, but something like Karapathy's LLM may run

A cluster of vapes, lmao

literal vaporware?

Vape-coding.

> a more challenging device

A Playdate

Aw man I would love to be able to use the crank to back up the agent dialogue, maybe make it branch forward in a different direction. Make the dialogue into a game

https://play.date/

pretty interesting, I don't own one, but it's possible probably with an emulator to figure this out, thanks!

Yes this has been my exact thought for awhile now.

Perhaps the real value with AI is going to come from...surprise here...a whole bunch of different low resource AIs networked together living and working and yes playing thru a networked "society"

Sounds kinda familiar doesn't it?

I made myself Remote Control Deluxe(tm). Which is essentially a remote control connected to my Claude and VLC.

I hold A to speak. B to cancel. Turn the crank really fast and really slow → send the prompt!

I still do use it sometimes.

The app talks via serial and WiFi.

vigorously turn the crank to speed up the output. ha.

perhaps the pi 1st/2nd gen?

I'd thought of the ESP32 devices but their RAM is too small

Pi 1 is too easy. My PicoLM [fork] already runs on it.

It also runs on MS-DOS 80386 2MB RAM, so...

And iPhone 1. And Digital UNIX.

Raspberry Pis in general are too easy for this task they don't even need modifications, older iphones seem more interesting due to some work on comptability.

Machines with unified ram work better? whats the bare minimum Ram you need? Maybe those old SGI or Sun machines could have a second life. SGI boxes made the Terminator come to life on the big screen so now their second act could be to help bring the real terminator to life :P

Yes please, my Indy, O2 and Octane2 are all sitting there waiting for some hot Terminator LLM action.

Try an iPod Touch 4th gen? IIRC it's the last model that doesn't support anything higher than iOS 6, so it would be fun to have a local LLM running in a native app that speaks skeuomorphism.

I have a basement full of Pentium 4-Core 2 Quad machines lying around. Has anyone strung together machines of this era to make something work? I always imagined that even if I were able to provide something like 4-8GB of ram per box the token speed would be measured in tok/hr or something.

the thing that kills them is the complete lack of any kind of matmul instructions. With my PicoLM fork it's possible to "play" GPT-2-AIdungeon on an iBook PowerPC 1.2Ghz with 1GB RAM (scalar instructions, as I said), but it's only for men of the more... patient variety.

Even a Pi 3 is a no-go for any model of consequence. At the very least get a Pi 4 or old ARM phones with NEON.

ipad Air I tried doesn't have matmul, but there are tricks you can do to work around it actually.

this kind of old processors are smth I want to explore honestly, I don't know if the processor capabilities will allow smth interesting to happen or not, but they're for sure on my target list.

I think realistically smth around 5tok/sec is possible with them for 2B models or so if you provided enough ram for streaming. worth a shot

I was looking for the lowest claim in this thread, as I managed to get a nano LLM running in the ≈3mb of SRAM of an FPGA with zero DDR in the loop at 20k tok/s, but 2mb on a 80386, great job.

Have you got a write up for it?

I ran YOLO on an FPGA for my bachlor's project degree project like 7 years ago, but this is even more interesting, and I think we can push bigger models with weight streaming. I still have an FPGA lying around.

I haven't written a new blog post for it yet but there's currently a demo live of my FPGA running bonsai 8b on the same FPGA here

https://www.mikeayles.com/blog/on-chip-llm-kv260/

Its right up against the wall of DDR bandwidth, so I'm trying every trick in the book to get the speed up

Decent!! Also I love the idea of the live chat, would you mind if I put a reference to it on my repo?

This would be fascinating. I've sometimes wondered how computing would have been different if we'd known how to build modern LLMs in the 2000s, despite the much poorer hardware (leaving aside how you could possibly train such a thing back then).

Wonder if its possible to solder on more RAM to the PS2 like you can with the Xbox.

Already done. Ofc you’ll be streaming the weights off the optical drive ;)

https://github.com/xaskasdf/ps2-llm

streaming weights of the CD drive should be top 10 blessed engineering hacks I've seen in the past 2 weeks.

A Speak and Spell that replies using the voice processor

Is it even still around? :D would be interesting if actually accessible idk

I got speak-n-spell voices working on arduino mini.

The only challenge in that case is ram (512, cpu is fine), but I think you should try getting something usable on the raspberry pi zero.

Once that's done it means people(or startups) can get an edge LLM subsystem on their projects for the zero or boards with more ram.

actually the pi zero is a nice target to play with next, will give a shot and report