tkellogg

how is this self promotion? What am I promoting? This post is literally rooted in curiosity, to the core.

Whelp, it took an hour for this to quickly rise on the front page and then get shadow banned. I guess that says something.

FYI https://mcp.so/ is the exact same thing as was posted. Not sure why they directed to the github instead of the actual site..

Somewhere in 2024 I noticed that "AI" shifted to no longer include "machine learning" and is now closer to "GenAI" but still bigger than that. It was never a strict definition, and was always shifting, but it made a big shift last year to no longer include classical ML. Even fairly technical people recognize the shift.

The R1 paper did it as well. Agreed, it's always very interesting.

it's just an example, but it's great to see smolagents in practice. I wonder how well the import whitelist approach works for code interpreter security.

Additionally, in the following days, I've seen evidence suggesting that the SFT part might not even be necessary. I'd argue that work wouldn't have happened if R1 wasn't released in the open.

Ah! My bad, I edited the article to simply quote Francois. Thanks for catching this, Simon.

Author here. I do believe it's going to be exponential (not yet), but that's out of scope for the article. However, if someone has a good explainer link for that, please put it here and I'll link it into the post.

I think their thought process is unconvincing, although I think they're probably correct.

A much better paper is, "Procedural Knowledge in Pretraining Drives Reasoning in Large Language Models". They took it much further and trained an LLM such that for every inference, they could see exactly which document from the training dataset it referenced to answer a particular question. ngl This paper AGI-pilled me.

https://arxiv.org/abs/2411.12580

Right, there's already some very encouraging trends (this study out of Nigeria). Clearly AI can lead to laziness, but it can also increase our intelligence. So it's not a simple "better" or "worse", it's a new thing that we have to navigate.

https://blogs.worldbank.org/en/education/From-chalkboards-to...

I added Phi-4 to my reasoning model collection because it seems to exhibit reasoning behavior, it stopped to consider alternatives before concluding. I assume this is related to their choice in training data:

> Chain-of-Thought: Data should encourage systematic reasoning, teaching the model various approaches to the problems in a step-by-step manner.

https://github.com/tkellogg/lrm-reasoning/blob/main/phi4.md

I feel like this conversation isn't complete without referencing Hillel Wayne's "Are We Really Engineers?"

https://www.hillelwayne.com/post/are-we-really-engineers/