theptip

Yeah, I think the multiple experiments muddy the water.

Early experiments show one-shotting direct weight interp, which is crazy to me. But this is IIUC only on early layers.

The later math result gave the LLM the ability to run forward passes and inspect activations, and as others have noted is more of a classic steering vector intervention. Impressive how low to the metal this one was, but it’s a lot less so than the openers. Plus, as you say this is just bugfixing a known flaw of repetition, by finding the “repetition direction” and top triggering neurons, then rewiring the neurons to fire suppressing repetition instead. This is not a new technique and doesn’t shed any mechanistic understanding on how maths works.

I guess the fundamental question would be, for what domains can we pretrain with “good views”?

Part of the magic of next token prediction is it works to extract intelligence from the unstructured web corpus.

Can JEPA formulate a similar scale training set of “good views”?

Isn’t big-endian vs little-endian a trivial example of how you can represent the same mathematical object (integers) in mutually-incompatible ways?

Maybe I’m missing the detail on how complex the mapping function can be before we say the representations are incompatible.

Haven’t thought too deeply about this, but as a physicist we’d do things like switch from Euclidean to Spherical coordinates or a completely different dynamical representation like the Langrangian, and this makes certain problems dissolve (or at least become much simpler to solve analytically).

Does it even make sense to claim there is a “platonic” / true representation? Maybe there is an optimal one for a specific task, eg representing a small map is great in the Cartesian frame. But you need a spheroid instead of a plane to accurately model the earth, so the latter is “more real” in some sense. But these seem like engineering tradeoffs of error vs compute and a given implementation could correctly pick either depending on its constraints.

I think there is a massive opportunity for this kind of feed disaggregation. Used to be the recommender and indexing of content was hard, now I believe we can likely build our own according to our own preferences.

Haiku/Sonnet (or even Jev) can do a lot of the filtering work to rank and extend an RSS feed list.

I think this gives us an opportunity to fully cut the cord from eyeball merchants who want us to maximally engage and therefore polarize their content.

The big remaining hook will be the social graph; I am interested to see how far the EU push for mandatory interoperability will go. Another angle I’d support would be mandatory pluggable standards for your own ranking algorithm, but it captures less of the win than full disaggregation.

> This gets to the same point of "If ChatGPT can make you a successful million dollar business in a few prompts why is OpenAI sharing it" paradox

I agree fully, to be clear o was using that prompt as a shorthand, I don’t think _you_ can type that prompt and get $1B back.

It can’t do this now, obviously. You need a lot of scaffolding and domain expertise to get the most out of these models. (Eg Harvey.)

But eventually, the labs can just run the business themselves.

I’ve long held that as we see models become more capable, they will be less available to the public. It seems pretty obvious that vertical integration on private models (perhaps a per-industry fine-tune, eg a legal model) licensed to subsidiaries will let the labs capture more value.

Why would you publish your model, enable competitors, and enable distillation of your hard-won intelligence? Only antitrust law can prevent this outcome IMO, the business pressure is extremely strong.

Congrats on the release - private repos are going to be huge. How do you see the positioning vs Buzz and Radicle? Do you see them doing anything better now, and are you planning to try to close any specific gaps?

From my very limited testing private repos were the big missing piece; interested in other areas yet to be filled in.

Would AI companies rather have $100/yr of license revenue or $480/yr of token revenue at 50% margin? Obviously the latter, by more than 2x? And even larger dollar profit from the higher sub levels you mention.

> Someone can always optimize a model down to running on a typical 32 GB M6 or M5 Mac. It doesn't need to be the best.

Two observations - one, there is some set of tasks where intelligence is already saturated, and sure, that threshold is rising by the month. So, sure, for some tasks you don’t need the best model. But there are other tasks that need models smarter than what we have now - like “make a cure for <disease>” that will yield billions of dollars of value, but likely require quintillions of tokens and higher IQ/token than anything we have now.

As long as we have non-saturated economically valuable tasks there will be incentive to train new and smarter models, and the frontier will always be on racks of GPUs not local distilled versions.

Two, it’s not clear that you can distill all the way down to 32GB even for the current frontier models. IIRC Karpathy has speculated that you can get an AGI into GPT-3 size, if you drop all the memorized facts and keep the reasoning core. But this is highly speculative! The best 32GB models right now are substantially dumber than even the large open-weight models, let alone the frontier closed models.

Perhaps I am missing something, what are you imagining of “a locally ran inference engine” that doesn’t already exist in a DGX workstation, Framework desktop, MacBook, etc?

It would have made HF safer, that’s for sure.

HF is a US company, so this is irrelevant to my point.

HF was not signed up to any of the (free to OSS) programs like Daybreak or Glasswing.

Perhaps the crux is: what is “value”?

If we mean dollar value then yes, hand crafted code may go to zero as nobody wants to pay a 10x premium for it.

If we mean (as I did) the subjective value to the creator themself, then there is no reason this has to stop. It will just make the same shift as farming to gardening.

You shouldn’t expect to get paid for this, or do it on your employer’s time, but you are absolutely free to do it in your leisure time, and perhaps just like Go and Chess, AI will make these pursuits more accessible than ever, to those who are interested in learning for fun.

No. It’s a healthy practice to build prototypes and spikes to resolve technical uncertainty. These should be considered disposable.

Sometimes people get attached to the prototype, and that’s counterproductive. The most common example is “nice job, the 2 week spike shows us that this project is not viable, we are canning it and moving on to a more impactful project”.

This is an underrated perspective. A common tactic I use with agents. Just building the slop prototype until it has enough fidelity to be obviously right or wrong can be way faster than sitting down, thinking hard, and planning in advance. This is especially true for background tasks, and IME is the key to productively parallelizing yourself without decision paralysis.

At some point token efficiency will bite, but even that is overrated; the metric you really should care about is:

impact / (salary + token spend)

> Throw nine autonomous agents at a single codebase without crystal-clear orchestration, and they will happily rewrite each other's work in a high-speed loop.

Mm, I think this is quite overstated in the era of Opus 5 / Sol.

What we actually see is that agents are much better at coordinating than humans. Numbers from OpenAI show 4x the agents gives 2x the speed.

Of course, for truly single-threaded tasks you can’t gain much by parallelizing, but ask Fable/Astra to chop a big task into small pieces, write the APIs first, then sub-agents take what’s behind the API, and you’ll usually get a much better speedup than if you asked humans to work this way.

In practice most big projects have crux choke points and then fanout migration/buildout phases.

Even on the cruxes, where you need a single decision, agents don’t have ego and so you can get speedups by eg. asking for 10 clean room implementations of the same problem, and have a judge model pick the best; this can let you go much faster, and no human team would put up with that collaboration model.

Missing the point. Which is, some people like the details, code as craft; others like the high level, product/business impact, systems, etc, code as a means to an end.

The profession used to have a need for both, now the “coding machine” and artisanal code crafter are trending obsolete in most companies.

It’s asymmetrical though. Google is terrible at product and moves glacially slow.

Anthropic can take the parts of Google it needs FAR faster than Google can do the reverse.

Yeah I’m really surprised at how little attention this is getting.

I’d say it’s more Notion than GDocs for now, but the direction is clear. Gobbling up all the SaaS in order of descending value to agents.

My bet is on promoting their internal Tasks representation to something more like a Jira/Linear.

But then obviously Slack is another one, and Duet is already starting to build a “Channel” abstraction, so making Duet multiplayer is a clear possible next step.

Already well documented that there is huge variance in inference serving efficiency; at every layer (model, software, hardware) you have opportunities for gains. Similar to how “the Chinese” haven’t undercut on cloud providers’ CPU offerings, there’s a very good argument that the hyperscalers will manage to keep their vertically integrated hardware/software edge here too.

I don’t really know where this commodification idea came from, maybe MBA types that don’t actually understand how cloud economics work, applying basic first principles analysis?

The operational engineering that goes into serving a frontier model is _highly_ nontrivial, it is not commodifiable. This isn’t a SPA application.

Hmm, most of this seems like it would be cutting edge in 2025. And why do I want to use Monaco? If I’m in the JetBrains ecosystem isn’t it because I prefer their IDEs over VSCode?

The versioned and linked spec feature sounds somewhat interesting though.

Yeah, process really needs to change. I don’t have an answer here, the only thing I’ve seen work is when a small group of engineers owns delivery of what would have been a team or team’s work, and can ship so fast that there isn’t really a “backlog”, you just build every idea and include or reject it.

As soon as you have many engineers using agents in the same codebase, the process breaks down, and you start stepping on each other’s toes, collapsing those velocity gains.

Maybe the “software factory” pattern will work, where agents just pull from the backlog, and so the team can all focus on grooming and disambiguating tickets, with the coordinator agent flagging conflicts and ambiguities between different people’s asks?

The only thing I’m confident of here is that a “h1 planning exercise” doesn’t really make sense in most cases; ~everyone should be shifting to building a lot of prototypes and iterating with customers, rather than trying to prioritize how to spend scarce eng effort. (That resource is no longer scarce.)

I think a lot of teams are going to rediscover (by necessity) the good parts of Agile, namely finding the domain expert and iterating with them to solve the actual problems; that’s your bottleneck now.

I think there’s a lot of value in a “stack skill” like this offers. I don’t personally buy that the scaffolding around “markdown source of truth” makes sense; I prefer to work as close to the agent harness as possible. An iterative conversation and then “cool, write a doc to memorialize and then build this” usually goes well for me, keeping the back-and-forth in context for as long as possible.

Since most of the comments are just throwing shade, here’s what this is actually doing:

https://rustoffrails.org/docs#the-stack-in-one-breath

>>>

Axum on hyper (the binary is the web server: HTTP/1.1 + HTTP/2, no app server or nginx needed)

Tokio · SeaORM (SQLite locally,

Postgres in production, same code)

Tailwind

a frontend mode chosen at birth:

htmx (default): Askama (compile-time checked templates) + htmx. No Node. Optional React or Svelte islands.

inertia: React or Svelte pages with Rails-style server routing, via a generated Inertia adapter. The default when you want a JS UI.

spa: a Vite SPA (React Router or SvelteKit in SPA mode) + a JSON API.

api: no UI, for mobile backends and public APIs (in progress).

> whether the complexity is "ahh we think this neuron does 100 seemingly unrelated things but we don't understand it" or "this actually is some kind of latent shared feature, but we don't know what the concept represents because the composition of the entire circuit is an arcane algorithm",

The claim is closer to the latter (monosemanticity) but also claiming that the concepts are mostly comprehensible and not arcane.

(The best arguments against this being a full solve of interp that I’ve seen go like: “we expected most of the complexity to live in the circuits, anyway”.)

Nick has posted some follow up to beware about over-claiming on this. You still need to interpret the circuits, as well.

Lots of work still to be done, this work just dissolves an early blocker (if it is substantiated).

Honestly I think this is a really bad look and sour grapes from the OP. Sapling is GPL. You open sourced it. What does “rip off” mean here?

If you think the license was breached and code was copied, that’s one thing.

But claiming ownership over an abstract idea in a project you open sourced seems epistemically confused to me. You don’t get to do that, there is no obligation from anyone to credit what OSS projects they drew inspiration from, as long as they comply with the license.

Having said that it’s nice to give credit to your inspirations, and Anthropic very clearly did:

> Source control for huge monorepos has prior art, notably Google’s Piper and Meta’s Sapling, so we were able to build Scion with their hard-won lessons in mind

This is just needless drama, nobody at Anthropic did anything wrong here.

This matches what I came up with on a small project. The biggest ones being screenshots and separated simulation.

Meta level, just push your agent to make things more legible, it’ll figure it out.

One other observation, remember that your agent is infinitely patient. So prompts like “build an interactive design doc with videos of each game mechanic in the style of Factorio dev diaries” can make reviewing changes easier. The general idea, all the small QoL dev tools that you don’t have time for, you can just spawn a side agent to build; if they are orthogonal to your main thread you can just progress them in the background.

One thing I think game dev has not (AFAIK) invested as much in is observability. There’s a lot of perf instrumentation available in debug builds, but things like measuring DPS, health, etc as an exported metric time series I have not seen before. I think agents will benefit from this disproportionately more than humans, eg for inspecting progression curves and balance.

Yes, easy assuming multi-generational arcologies.

There are plenty of designs that get you to 0.1c or above if you are Kardashev 2, things like light sails powered from the Dyson sphere.

Nothing plausible for anyone living today AFAIK.

I’m in the same boat as you, but sadly I’m not convinced of this:

> that is most people's experience with LLMs

I think the median LLM user in high school, for example, is mostly using it to avoid writing essays themselves.

I don’t have great evidence here, mainly just the plummeting exam scores. It could be that kids are also using llms to learn about the things they do care about. I’m skeptical.

Right, this would be “using it in a way that increases not decreases human knowledge”

I’m fairly confident that many won’t (immediately) adopt such techniques.

I hold out hope that there will be new ways to make educational content “addictive enough” via personalization that AI is on net a positive to education. I don’t think this is necessarily the easiest or first product configuration.