visarga

Concepts emerge as good compressions of data, or repeated patterns, or recurring distinctions that prove useful to name and recognize.

Good point, I think platonism is wrong, it puts concept before execution, and often removes the lineage behind an idea and sees it as standing on its own. We like to imagine tings without their costs, but cost pressures are what make those things what they are, such as what model works better for a problem.

The new copyright protects vibes

I think it is a good approach for this year, but it misses something important - how do you know the code still reflects your intention after steering for weeks and months, across many sessions? Messages flow in the session and zoom out of view, and across compactions/new sessions they get forgotten. To solve this, I log every user message in a chat log, and review it from time to time to distill the latest form of user intent.

Another kind of review not mentioned here or in Anthropic's article is the plan review. I generate the plan for some stretch of work, then have the plan (saved as a file) reviewed by one or more agents. This checks the plan against project constraints before executing it.

If we can solve testing then coding is almost fully automated. The only remaining problem was always trust, why should we trust that code? If you read it manually then it's like walking a motorcycle. If you don't, then you need some way to derisk it. There are many ideas flowing around about how to do this, the OP and Anthropic showed their takes. I think the approach is project specific though, no general method.

It would not be inside Wikipedia, but separate, as a collection used in the training corpus.

A billion humans use AI daily, they write about 0.5T tok and read about 2T tok (estimated). In a week they amount to more than the whole training set of GPT-4. Human language is being shaped back by AI language.

I think this is exactly what we should do. Send agents to collect evidence and write deep research reports, then train on those reports. DR reports are derived from hundreds of sources and can map out the distribution, what are the opinions on a topic, not necessarily what is the one correct answer. Agents could write one piece about every human, place, topic, event or entity there is under the sun. They could also try to generate the same without sources (closed-book mode) and report on LLM biases on that question.

We don't have to fall off a cliff to learn to drive on a mountain road, we can imagine the consequences for cheaper. If we didn't have imagination life would be much more dangerous.

> is just profoundly goofy shit

Why not expect it? It's what most model releases do.

The SOTA model dilemma is that you have to pay more and more to train models that focus on less and less of the AI task market, because 95% of tasks can be done by cheap small models.

The context is that which can pay - a human or bank account. It's the liability sink which is immune to AI, nobody cares if your AI failed or you.

I think the article is insightful. I want to contribute my perspective:

1. when an AI starts working on a task, it is within a human context, someone pays and decides to assign that work to AI

2. while it performs work on the task it continuously needs to interact with the context in order to act and observe the feedback

3. at the end of work, the outcomes remain in context, including gains, losses, costs, risks and liability.

So AI is great but can't do the beginning, middle and end of a task. They belong to context, which is us, the AI users. Context is maximally distributed, every human, team, or company that uses AI is a context, they are relational and indexical. You can't have my context, for example - you can't eat for me, or learn for me. Some things must loop back. Especially consequences of AI work. That is irreducible, AI can't improve to grow a skin like us. Its own skin is user base gain/loss, competition with other AI providers and dancing for investors. Every AI usage context is not theirs and they can't own it.

What happens to companies is that their investors expect AI usage already, their competitors use it to compete with them and fast follow, and customers use AI agents to arbitrate their buying opportunities. So even if they did nothing AI related they find themselves in a new world where.

1. anything visible can be easily imitated - code, text, media, and AI agents - I call it the fast follow effect

2. any apparent arbitrage opportunity will be quickly crowded by fast followers removing profit from it

3. AI it self is competitive and the whole stack is fungible - models, harnesses, tools, they can all be replaced easily, what matters more is the project context data which is usually a git backed folder on local disk

So my theory is that AI won't make anyone "rich". Instead we have a musical chairs moment. AI removes distinctiveness from everyone of us. Every time I use AI for myself it is just great and amazing. When others use it, they make our own work seem little and not as impressive. Companies are caught in a Red Queen effect, have to work harder just to stay in place.

And here I find the answer to what will we do with all our free time - compete, run for our own musical chair. We need to find our own niches again, they are now under rapid change. We will invest as much energy as needed to do it, we are ready to pay a lot of effort to differentiate back. But we can't predict it just as you can't predict where the ball in a football match will go 1 minute later, it depends on what every other player is doing.

You can lessen the human review burden by pitting multiple agents against each other. Ask 3 agents to implement the same solution, then set them up against each other in differential tests. This can find bugs in them or clarify the spec. When tokens get cheap it is worth having multiple replicas, taking advantage of different programming languages, libraries and agents. You don't keep one version, you have a population, and it tests itself. It's easier to implement again than to imagine testing scenarios anyway.

You can "solve" 1 and 2 by N-version programming. Have the same code implemented twice, or more times, you get free error testing by fuzzing or running replicas in shadow mode. In order to make replicas maximally different choose a different language, framework, agent harness and LLM.

There remain some nitty details related to tracking randomness and choosing what interfaces to log. But even having your code testable by replicas would help organize it better and expose different assumptions made by the language or library stack.

When you get different outputs from replicas it can be 1. one of them has a bug, or 2. the spec is unclear -> fix the bug or the spec. So this method catches both sides. It also might be faster than traditional code based testing, you can build replicas in parallel. A replica is an open ended test, unlike coded tests it can be tried on open ended input values.

The core idea is that it is easier to implement twice than to imagine bugs. We already solved implementation, but bug finding means running code in your head, it touches the halting problem issues - can we really imagine behavior from code? I don't think we can do it well. That is why we should just make replicas instead, tokens are getting cheaper and cheaper by the month.

> The second form of regulation should be on excessive energy consumption. The US Department of Energy needs to declare energy to be a vital human resource, and exorbitant uses for computing to be curtailed.

If you allow AI to use as much as it needs we get much higher capacity and cheaper energy down the line. Growing pains.

I see AI like the coming of a baby in the family - suddenly you need to get a larger house, baby room, ensure babysitting and education, an effort for the whole family. This is the initial cost, which is high for both a baby and AI.

In my own words - you can't calibrate confidence of a model without a calibration set. Confidence is relative to some labeled dataset, it is established on a set of buckets or intervals, and is population based, something you can't check for a single prediction. Assign per case, verified per population.

Could be the missing link in the sovereign stack - an agent with a search stub library running locally with higher privacy than using dedicated search providers.

I think computers, internet and AI - all of them have impact - but they consume it by converting it into new structure. The benefits are in the infra not handed out to us individually. We might not see large changes on individual level, or the displaced work is replaced by management work for the automations and raising baselines push us to do more. We now have data centers, fiber backbone, GPUs, models, apps and open source projects, yet we still only have 24 hours a day.

Why can't we simply save search stubs, short documents a LLM can retrieve, use them to route to various websites, resources, feeds, forums, blogs, other specialized search engines, APIs? If you host a 30GB model, then you can use 30 more GB for a compressed search stub index. The entry points for any domain are more stable than the pages. You would use an agent to navigate the resources as needed, fetching fresh data.

It's not quite search, but it can replace most search. It is a router for search, where a local agent performs the live interaction. The local agent does the live work, this maximizes privacy too. When your local stub index doesn't cover the topic of interest you can use any coding agent claude/codex/gemini/etc to research and generate the missing stubs.

You don't need to read my message if you think you know what is the best formula for your AI usage. Whenever someone complains to be "tired" of reading some online message I take it as an act of verbal aggression. I rarely disclosed my formula online, and probably never will again after such reaction. Do you want to regulate HN? Was my message off?

Unfortunately the gains of automation will not become our post scarcity utopia, instead they will be consumed by the new AI overhead and we remain in a regime of scarcity. AI will reshape all our processes and societies, that is where the gains will go.

All good ideas, I have been doing something similar for half a year. The most important thing is to reduce bugs and ensure the code does what you wanted. Writing code tests, logging, interacting with the running app are essential. For intent realignment I like to have the agent re-read all user messages posted in that project. It only takes a few minutes even on a 10,000 message project. Then use this fresh derivation of user intent to check code and tests. For long spans of work I prefer to generate a todo list which remains as a document in the project. Each closed gate tracks feedback from the agent on what happened. This task file is also reviewed twice - once before implementation and once after. So the formula is - review agents - tests - user message logs.

Sure there is, you said it yourself. You can make all those changes because it's easy, the lock-in effect is waning. We can move around and take better deals and choices.

This is actually a scary side of AI for providers: we use our own AI to research better choices and escape dark UI patterns and vendor lock-in. That information asymmetry and the difficulty of leaving a provider used to be their moat.

On a system level I think AI code is more like biological diversity, it hedges against monoculture risks. We don't all fall down at the same time if we don't replicate the same code.

Yes, I realize the NS singularity does not have practical usage yet, but it was a Millennium Prize problem. It's funny how Vi discarded it

> or input generated by them as derivatives of human input

Trained mostly on their own data, not ours. The mix changed, from 10% to 90% synthetic data. RL training is now the largest part of the process. The AI work is searching, and I don't think we can claim this process to be credited to us anymore. The origin point is, but not the journey.

> but they can't do on their own what we did not demonstrate how to do first, because they are tools.

Funny thing to say here, where an agent swarm (1M hours agent time) just did something no human did. If someone else solved NS blowup before OpenAI we would have heard their claims already. And we did hear partial claims, "they rushed in the same direction" kind of claims, not "they copied my fully developed idea". I think this is Move 37 moment for math, when we realize we might not be the top anymore.

Any expensive search in an exponential space meets cost very soon, but Vi is a Platonist, she thinks AI generate meaningless tokens, such as enumerating all proofs by LEAN. No, that search is interacting with its own costs, it is not platonic search scaled to infinity.

About three quarters of Finland is forest, the highest share in Europe. Cutting down a small area is blown out of proportions, they could of course plant trees to offset in another place. Put every datacenter on Earth together and the land would hold maybe 20 million trees, about 1 in 150,000 of the world’s trees, or half a day of global deforestation.

> now LLMs just buzz through the file changing thousands of lines and we can't keep up. It's still valuable to deeply understand parts of a program, but we don't have any tooling that helps us do that.

I do it differently, I focus on better recording what the user wanted, the so-called "user intent". To do this, I record all messages typed by the user since the start of the project, whether 3,000 or 10,000 messages. An LLM can churn through them in 10 minutes and derive a fresh, up-to-date interpretation from the raw data. This can be used to judge whether the implementation has diverged from the intent, or, in other words, to realign the code and tests. The messages the user writes are usually designs or corrections, a very rich, compact signal. If the user struggles with something, it could result in a tool, a skill, updates to the project docs, or new tests.

When anyone can do these activities they will no longer confer distinction. So we are now motivated to find other ways to differentiate ourselves. This powers the next round of activities - our need to stand out by something.