Re:Microgpt
by WithinReason on Sunday March 01, 2026 @04:21AM
(#47205095)
I made an explicit reverse pass (no autodiff), it was 8x faster in Python
Can you share a link?
Re:Microgpt
by WithinReason on Monday March 02, 2026 @11:54AM
(#47220542)
https://www.ideone.com/VAz4Nn
Doesn't run inside IDEone due to the external download link, but you can copy&paste the code over
I made an explicit double-reverse pass (no code!), it was 80x faster in my head!
"I've got an ipod -- In My Mind"
Re:Microgpt
by WithinReason on Monday March 02, 2026 @04:26PM
(#47224335)
code here, it's just not interesting to look at:
tradeoff worth naming: you avoid the autodiff graph overhead (hence the speedup), but any architecture change means rewriting every gradient by hand. fine for a pedagogical project, but that's exactly why autodiff exists.
24x speedup (over 10x already) and similar loss profile (for c++ version, optimized by claude): https://gist.github.com/freakynit/3982eab8413a89941bd0018e63......
2x the number of lines of code (~400L), 10x the speed
The hard part was figuring out how to represent the Value class in C++ (ended up using shared_ptrs).