sasjaws

The idea i tried to express was purely the loss function thing you mentioned, and how both tasks (1 vs 2 vs n) lead to identical training runs. At least with nanogpt. I dont know if that extrapolates well to current llm internals and current training.

No blog post, my llm expert friend told me this was kinda obvious when i shared it with him so i didnt think it was worth it.

I can tell you how i got there, i did nanogpt, then tried to be smart and train a model with a loss function that targets 2 next tokens instead of one. Calculate the loss function and you'll see its exactly the same during training.

Sibling commenter also mentions:

> the joint probability of a token sequence can be broken down autogressively: P(a,b,c) = P(a) * P(b|a) * P(c|a,b) and then with cross-entropy loss which optimizes for log likelihood this becomes a summation."

Hope that helps.

This is about reasoning tokens right? I didnt mean that, nanogpt doesnt do that. Nanogpt inference just outputs letters directly, no intermediate tokens.

Thats actually an interesting way to look at it. However i just posted that because i often see articles expressing amazement at how training an llm at next token prediction can take it so far. Seemingly ontrasting the simplicity of the training task to the complexity of the outcome. The insight is that the training task was in fact 'predict the next book', just as much as it is 'predict the next token'. So every time i see that 'predict the next token' representation of the training task it rubs me the wrong way. Its not wrong, but misleading.

I didnt mean to suggest that is how it 'thinks ahead' but i believe you can see it like that in a way. Because it has been trained to 'predict all the following tokens'. So it learned to guess the end of a phrase just as much as the beginning. I consider the mechanism of feeding each output token back in to be an implementation detail that distracts from what it actually learned to do.

I hope this makes sense. Fyi im no expert in any way, just dabbling.

Thats the one, lots of fun and a great entrypoint for experimentation.

A while ago i did the nanogpt tutorial, i went through some math with pen and paper and noticed the loss function for 'predict the next token' and 'predict the next 2 tokens' (or n tokens) is identical.

That was a bit of a shock to me so wanted to share this thought. Basically i think its not unreasonable to say llms are trained to predict the next book instead of single token.

Hope this is usefull to someone.

Personalized audio streams for language learners. Ideal for during driving or while doing chores.

https://listen.longyan.io/

At the intermediate level lots of learners struggle to find suitable content that matches their level and interests, more than a few learners turn to notebookLM podcasts to provide that, but that's a bit of a hassle to set up. So I built a platform that generates and manages infinite and shareable streams around your interests or specific vocabulary. It also provides live interactive transcripts (karaoke / teleprompter style) if you need it.

Core features work but still rough around the edges. Happy to help you out with any issues you encounter, languages to add, feature requests etc...

I've added support for Lithuanian and created a stream about version control for you to try it out. Just 'select language' -> Lithuanian -> Play

If you find it useful, you can register for free and create new streams on any subject. Send me a mail on [email protected] if you'd like more stream/content quota or if you want to try the Anki thing, I'll gladly set it up for you.

I'm building a service that generates audio streams about subjects and vocab of your choosing, currently notebookLM based. If you have intermediate listening skills its pretty useful for deepening regular vocab and acquiring specialized jargon.

I dumped my 400 hardest recurring anki words in it and listen to the stream whenever doing chores or driving. Then sync with my deck again after a while.

Can you help me out and give it a try, you seem like the target audience and i'd value your feedback. If your target language is not available or want to upload an anki deck I can help you out.

https://listen.longyan.io

Thanks for having a look, I actually started out from traditional characters, but once I realized >90% of the students only do simplified I switched.

I also tend to believe to just convert between them is not the best approach. Better to find different content for both. If student wants to learn traditional script, they usualy want content from Taiwan and not from China, and the other way round.

I'm building a reader app that tries to solve this exact problem by providing a range of gradually simplified versions of each article to match your proficiency. So you can stay in the sweet spot, or work your way up version by version.

If your target language happens to be Chinese then you can give it a try at https://reader.longyan.io/landing

No login required, love your feedback.