itkovian_

Also no doubt they’ve coined a term with weight space programming and this will expand into an active sub field

This was shocking to me; but on reading it makes far more sense.

- the primary experiments are on small policy networks - they are compressed dynamic system models. The observation and action space can be read off. It’s not surprising to me that these types of networks can be used to identify the env they were trained on

- the exps on actual llms remove looping behaviour and that’s it - from article “but it could not add an ability the model lacked”

I’d say this is fortunate because if llms could do the latter, rsi would be immediate. It may be possible we simply need bigger and better edit models, but at least from the experiments there doesn’t seem to be any indication that is imminently about to work.

Conceptually incredibly interesting non the less.

It’s not professionals that need to something- it’s absolutely critical we have some kind of credibly neutral body providing the actual models. What’s not clear at all is how to do that since it’s such an expensive activity.

For now we have a great situation with the open weights, but there is absolutely no guarantee that continues, and more the opposite- they may stop.

Because if you don’t open weight you can dictate pricing as you’re the only party that can serve the model. Also attracts users to your platform rather than inf providers.

This is a major, major datapoint. That’s at least an order of magnitude higher margin than literally anyone is discussing. The entire ai discourse currently assumes very small margins on open model inf. And by this post it’s more like 1600% (h100s rent for $2/hr - and id guess deepseeks costs are far below that)

This is evidenced by the prices on every large-model capable device/node increasing significantly over the last year. Demand is far exceeding supply.

Your just missing many things and so have an incorrect picture of the situation - k3 is not on par with fable or astra. Closed models remain far better than best open weights at least today. - compute is not just used for training, more and more is inf - even in training you don’t do one run, you do many. Final run is a small portion of total compute.

The world is extremely compute constrained currently, like extremely.

Well I mean Verda is significantly cheaper than the hyperscalars for the exact same device so that’s one thing.

Maybe one argument could be there’s very little value added services required over the raw compute in this new world, and they’re structurally setup to capitalise on that whereas hyperscalars are not.

I’ve felt for a long time that the things I loved when I first started programming aren’t really there anymore. This comment made me realize that world truly is completely gone.

I can’t count the number of times I’ve heard variants of ‘they’re losing money on every query’ and ‘I’m getting 10k worth of tokens for $200’ over the last year. People clearly believed serving margins were -ve

What are you talking about - I feel like we’re living in parallel realities. If I had to go back to opus 4.5 tomorrow I’d be hugely upset and significantly slowed down

The author is trying to make the point that there are courses of action available to Dario that would achieve his stated goals but would result in Anthropic having very little impact, control or influence. And that because he’s not doing those things, his motivations extend beyond the stated goals.

It’s a good point and not exactly made subtly but seems completely lost on most comments here.

I can’t stand it. Very engineering-y over specified formal language around a complete lack of core understanding. Is damaging other people read this and try to learn things from it.

Causal masking allow model to learn implicit positional embeddings. The meme that a transformer block is permutation invariant is not true.

Intrinsic motivation/curiosity driven exploration/novelty seeking feels like it has a breakthrough paper waiting. If someone could get those methods working for LLMs, we start getting things like move37 but in math proofs and then everything else

the difference is all the current capex is going to durable, hard to get physical assets + things like PPAs. In your online shopping analogy, the hyperscalars are acting like Amazon in 1998

I just don’t understand this view. This is the most significant technology ever developed. The uncertainty currently is whether it 1) has massive impact, completely altering society and the making world significantly significantly better or 2) if we go into a fast takeoff/rsi loop. Personally I’ve always been highly skeptical of the later, but that seems like a genuine possibility now. It’s not ‘are we going to be able to generate 10% roic on compute’ the answer to that is yes.

‘640kb of ram should be enough for anyone’

Contrarian view; I think he’s right. Many of these ideas it’s almost shocking how many you can find sketched out in his old papers. To the point where I think it’s very wise to read all his work to see what hasn’t showed up yet but likely will. Artificial curiosity for example.

You can’t run a closed llm locally. Strange to frame the dichotomy as between local and open. One begets the other.

The entirety of Anthropic believe ai is going to eat everything, not just software, and result in major societal disruption within a year. They do not have a sliver of a doubt on this. Article has no idea, is completely wrong.

This is called linear mode connectivity and seems to work for almost every large model. So well that in most cases it’s an explicit part of the training process; do many training ‘branches’ then merge then continue.

It is not understood why it works so well.

Projects like pluralis agora solve this problem. Really what you want is the model to be collectively owned and governed, not local

People forget the people in charge of these companies are some of the smartest people out there full stop. Far more shadowy strategy/things like this going on than people think.

What are the odds this is partially them making the point; you were all complaining about monitoring/access/safeguards: remember we don’t have to give this to you at all. And using a us gov letter as justification for that.

In many cases the GPs are not, at least nowhere near as much as you’d think. Obviously there are power laws here as well. Non partners, forget about it.

The other thing is it is maybe the most nepotistic industry out there. Which somewhat makes sense given the actual job is so relationship based.

The extreme privilege of being forced to pay a major portion of all income you make, regardless of where you earn it, to the us gov indefinitely. And they make it hard for you to apply to do this. Crazy.

The US isn’t what it used to be. It’s definitely not the best place in the world to live for quality of life, on basically any metric.

The requirement of being permanently obligated to pay us taxes on global income, if you have any kind of global mobility, is not worth it when you look at the situation objectively. The US is the only country that requires this, and signing up is voluntarily.

So while US immigration continues to act as though people will jump through any hoop they put up in order to be granted the extreme privilege of being able to live in the country indefinitely, it’s worth realising it’s not the 70s anymore and thats a goal many people are no longer optimizing for. In fact the opposite - the most talented people I know are all planning their lives to not settle long term in the US.

The other thing people don’t understand is exponential curves are self similar. The start of an exponential looks like an exponential. People always look at and think ‘well that’s it it’s exponential now, have missed it, can’t sustain’. Nope.

Good example of this is number of submissions to neurips/icml/iclr. In 2017 that curve was exponential.

The argument is that there is no incentive to carefully review a paper (I agree), however what used to occur is people would do the right thing without explicit incentives. This has totally disappeared.