TinyStories: How Small Can Language Models Be and Still Speak Coherent English? (2023)

Re:TinyStories: How Small Can Language Models Be and Still S... by nerdponx on Thursday January 02, 2025 @03:40PM (#42578612)

Lest we forget that this stream-of-consciousness confusion was state of the art just a few years ago.

It makes sense if you think about it: a small model's "internal state" isn't rich enough to keep track of whatever it was supposed to be talking about.

It makes me think that the reason LLMs need to be so large is that the internal state needs to be bigger than a typical human "idea", whatever that might mean.

[ Parent | on HN ]

Re:TinyStories: How Small Can Language Models Be and Still S... by acchow on Thursday January 02, 2025 @11:33PM (#42582459)

The way we do LLMs now is that the program and the data are one and the same. The program mutates itself as it "executes". This is probably also how the brain works since there is no hard separation between "memory" neurons and "data processing" neurons. (biology has no hard separation in general).

[ Parent | on HN ]