attentionmech

why don't they just block the obs project and let users install it in unofficial manner while removing themselves as middleman? I mean, they have certain let's say guidelines but why go about enforcing them in this weird manner.

saw that video just now, thanks for this.

he has earned it haha.

I definitely avoided ML for years just due to math. But having a chatbot who can explain math with examples in any style you want defintely changed my opinion about math and ML in general. A big barrier to math is how it's written imo and not explained in a fun way with lot of examples. I certainly don't have a mathy brain, but I do get things when explained with examples (and certainly find it hard to come up with my own examples while fighting with the symbols).

Will checkout jeremy's lectures. I actually use his fastbook notebooks a lot to self-study.

Karpathy's style, for me is more like at the right abstraction to bring out curiosity in me towards the subject. After watching his lectures, i go on to more materials generally, and never really stop there.

So much respect for this guy. He is like Neo of the matrix, bridging the gap between humans and machines. I have so far learned the following for free from his repos/videos:

1. minGPT, nanoGPT (transformers)

2. NLP (make more series)

3. tokenizers (his youtube)

4. RNN (from his blog)

There are many domains which don't have a karpathy and we don't hear about them. So glad we have this guy to spread his intuitions on ML.

agreed. that's err on my part to mention it like that. more evidence suggest that they were working on similar stuff but now the cat is out of the bag and open source got a win.

people already did: https://x.com/karpathy/status/1884678601704169965

This is cool, and timely (I wanted a neat repo like that).

I have also been working from last 2 weeks on a gpt implementation in C. Eventually it turned out to be really slow (without CUDA). But it taught me how much memory management and data management there is when implementing these systems. You are running like a loop billions of times so you need to preallocate the computational graph and stuff. If anyone wanna check out it's ~1500 LOC single file:

https://github.com/attentionmech/gpt.c/blob/main/gpt.c

I love this paradigm of reasoning by one model and actual work by another. This opens up avenues of specialization and then eventually smaller plays working on more niche things.

I found the following thread more insightful than my original comment (wish I could edit that one). A research explains why RL didn't work before this: https://x.com/its_dibya/status/1883595705736163727

people are doing all sort of experiments and reproducing the "emergence"(sorry it's not the right word) of backtracking; it's all so fun to watch.

Yea, they might be scaling is harder or may be more tricks up their sleeves when it comes to serving the model.

Plus, the speed at which it replies is amazing too. Claude/Chatgpt now seem like inefficient inference engines compared to it.

Do you think this feature i.e. 'finding smaller chunks easier to solve' comes out from the dataset these are trained on or is it more related to architecture components?

Most people I talked with don't grasp how big of an event this is. I consider is almost as similar to as what early version of linux did to OS ecosystem.

If you check failure section of their paper, they also tried other methods like MCTS and PRM which is what other labs have been obsessing about but couldn't move on from (that includes bigshots). Only team which I am aware which tried verifiable rewards is tulu but they didn't scaled it up and just left it there.

This sort of thing imo is similar to what openAI did with transformer architecture i.e. google invented it but couldn't scale it in the right direction and deepmind got busy with atari games. They had all the pieces still openai could do it. It seems to be it comes down to research leadership in what methods to choose to invest in. But yeah, the budgets big labs have, they can easily try 10 different techniques and brute force it all but seems like they are too opinionated in methods and less urgent on outcomes.

[paper] https://arxiv.org/pdf/2501.12948 [tulu] https://x.com/hamishivi/status/1881394117810500004

That's nice explanation. Is there any insights so far in the field about why chain of thought improves the capability of a model? Does it like provide model with more working memory or something in the context itself?

the tulu team saw it. but, yes nobody like scaled it to the extent deepseek did. I am surprised that the faang labs which have the best of the best didn't see this.

idk what i am doing but i am hooked on it. it's like as if it's directly interacting with dopamine of my brain.

It's commutative. Happiness also doesn't buy money.

A problem is only a problem if you don't want it.

The author got "enburdened with a shitload of money"

"Before enlightenment, chop wood, carry water. After enlightenment, chop wood, carry water"

You do what you were enjoying before, just now you don't need to get paid in money for it. May be you will get paid in fame, noble prize, smiles of people whose life you impact etc. The money you have can't buy those directly but only when you put effort.

They are currency of reputation and status. If you have enough stars, you get invited to private parties with elites. (I am just joking, they are bookmarks who got famous)

wow, this RWKV thing blew my mind. Thank you for sharing this!

default to git --shallow in the cli can be one option here.

May be with these rules: - Per user account we only count one clone - We don't count anonymous clones

But I agree it's not like this is also without any issues

I think it's like a "upvote" thing which shows whether historically users have found the repo interesting. Even if you hide stars, there needs to be a way for the collective hivemind of github users to help each other with what repos are high quality or not right?