syllogism

It's obvious that recursive self-improvement has been happening and accelerating quickly for some time.

AI coding assistants make it much easier to conduct experiments and improve AI models, which in turn improves the coding assistants etc.

If there's a linear relationship between the weights and the output, it's not so hard to read the weights and reason about what's happening. If you read the reasoning traces for how the models do those tasks, you'll see that they rely on those direct relationships.

The same could never work in a larger network. It's not about just capacity to feed in the weights, it's about the complexity of the relationship. There's no demonstration here that an LLM can do that, and it doesn't really make sense to want it to. It makes much more sense to author code to analyse and modify than it does to do it directly.

I don't understand how anyone can read the abstract and think it's "ridiculously clear" that the "Just by looking at the raw weights," doesn't apply to the Qwen experiment.

Here's the work that shows how rule-based logic can be compiled into transformer weights:

* Thinking Like Transformers” by Weiss, Goldberg & Yahav (ICML 2021).: https://arxiv.org/abs/2106.06981 * Tracr: Compiled Transformers as a Laboratory for Interpretability https://arxiv.org/abs/2301.05062 (code at https://github.com/deepmind/tracr)

So it's not really an "LLMs can modify weights" thing. Rather it's "LLMs can write programs that can write or modify weights". Indeed LLMs can write many programs, it's impressive.

If you read comments about the paper it's pretty obvious people are misled. People think that the method read in Qwen weights directly to improve it. That's not what it did --- it authored a program to analyse and fix the weights.

What's confusing is they do have experiments that read raw weights, and the abstract is misleadingly written to make it sound like that's how the Qwen experiment works.

They make it sound like they improve Qwen by reading raw weights. They don't. In fact the experiments that use raw weights have very little to do with the experiments that work on larger models.

It released with unbearable keyboard controls instead of the familiar point-and-click. If I remember right a remaster eventually came out that allowed mouse control.

I agree that I hated the 3d though. This was a bad patch for games when everyone wanted to go 3d for the sake of it, even though it looked and played worse.

You can't take any statement like this remotely seriously. We live in a world where NSA officials can testify before congress that they don't "collect" data, because that's true under some baroque definition of "collect" that they invented and didn't tell anyone else about.

Similarly you have no idea what definition OpenAI intend for terms such as "specific user data", "accessed" etc. And we have no idea what non-excluded possibilities actually did happen that they simply omit from their statement.

In practice OpenAI and many others have created a situation where they're actually unable to make any credible denial of anything really.

If you're submitting an eight page conference paper you're very often hunting runts (short lines ending paragraphs) and rewording for better justification, especially on the first page and even more so in the abstract.

First-page space is at a premium because you want the reviewer to scan the introduction well, and it's great if you can fit a key figure that explains the concept on the first page.

The game's probably very different now that there's such a flood of submissions but when I was reviewing I always saw messy alignment on the first page as a sign the paper was probably rushed, and usually I could see that elsewhere in the paper too.

The optimal amount of stress about that stuff is not zero, and most teenagers are going to be desperately biased towards "none of this matters, just peermax".

This interview feels weird to me, maybe because there's lots of places where I'm thinking there's an obvious idea that isn't raised.

Like, it seems obvious to me that sleep has a higher opportunity cost in humans than most other animals. So naturally there's more evolutionary pressure in humans to sleep more efficiently.

Is this actually true? Dunno! But it was really weird that it doesn't get brought up.

It comes up because people are mistakenly conflating consciousness with moral personhood.

People want to be talking about whether AI suffers in a morally meaningful way. In non-human animals this debate is often centered around the question of whether the animal has conscious experience, because there's little doubt that much of the emotional and experiential systems are shared.

The analogy goes wrong with AI, where definitions of "consciousness" would seem to apply in the sense that the model clearly has a category for itself in its world model, feeds back on its output, etc. However the analogy between how it works and anything we would recognise as emotion or suffering is extremely strained.

The solution is to just focus on ths question of what we really mean when we think of morally relevant suffering. It's a much clearer question than "consciousness" and it sidesteps the problem.

You're going to make life much harder for yourself, not easier, because you'll still need German legal advice but now you need an expensive multi-national lawyer/firm. Anyone cheaper will refuse to touch it.

Germany cares about where the management of your company actually happens, not just where the entity is incorporated. So you're not going to avoid German bureaucracy, it's going to be worse not better.

Just keeping a lockfile and updating it weekly works fine for that too yeah

Maintainers need to keep a wall between the package publishing and public repos. Currently what people are doing is configuring the public repo as a Trusted Publisher directly. This means you can trigger the package publication from the repo itself, and the public repo is a huge surface area.

Configure the CI to make a release with the artefacts attached. Then have an entirely private repo that can't be triggered automatically as the publisher. The publisher repo fetches the artefacts and does the pypi/npm/whatever release.

The corruptions of this administration are legion, but this isn't one of them. Unless you can point to something Lutnick did to create this outcome, I don't see how he had a better view of the whole thing than anyone else.

Government's free to not like the terms and go with another provider. That's whatever.

Government's not free to say, "We'll blow up your business with a false accusation if you don't give us the terms we want (and then use defence production act to commandeer the product anyway)". How much more blatantly authoritarian does it get than that?

If you think a blue government would even consider threatening to falsely accuse a company of being a supply-chain threat in order to gain leverage in a contract negotiation, you're insane. There's nothing remotely normal about this, it's not something you see in any western democracy

The actions of the US government here are openly corrupt.

The point of the supply chain risk provisions is to denote, you know, supply chain risks. The intention is not to give the Pentagon a lever it can pull to force any company to agree to any contract it wants.

Hegseth doesn't even pretend that Anthropic is actually a supply chain risk. The argument for designating them so is that _they won't do exactly what the government wants_.

People use the term "fascism" a lot and people have kind of tuned it out, but what do you call a government that deals itself the power to compel any company to accept any contract, and declare it a pariah on thin pretext if it objects?

By taking the deal under these conditions OpenAI is accepting this. They're saying, "Well, sucks to be them, life goes on". They're consenting to the corruption and agreeing to profit from it. But they'll be next, and if the next company in line has the same stand then yeah, the government can force any company to do anything. There's nothing normal about this.

I don't understand how any sort of deal is defensible in the circumstances.

Government: "Anthropic, let us do whatever we want"

Anthropic: "We have some minimal conditions."

Government: "OpenAI, if we blast Anthropic into the sun, what sort of deal can we get?"

OpenAI: "Uh well I guess I should ask for those conditions"

Government: blasts Anthropic into the sun "Sure whatever, those conditions are okay...for now."

By taking the deal with the DoW, OpenAI accepts that they can be treated the same way the government just treated Anthropic. Does it really matter what they've agreed?

You should quit because the only reasonable thing for your leadership to have done is to refuse to sign any agreement with DoW whatsoever while it's attempting to strongarm Anthropic in this fashion.

It doesn't even matter if OpenAI is offered the same terms that Anthropic refused. It's absurd to accept them and do business with the Pentagon in that situation.

If you take the government at its word, it's killing Anthropic because Anthropic wanted to assert the ability to draw _some_ sort of redline. If OpenAI's position is "well sucks to be them", there's nothing stopping Hegseth from doing the same to OpenAI.

It doesn't matter at all if OpenAI gets the deal at the same redline Anthropic was trying to assert. If at the end of this the government has succeeded in cutting Anthropic off from the economy, what's next for OpenAI? What happens next time when OpenAI tries to assert some sort of redline?

What's the point of any talk of "AI Safety" if you sign on to a regime where Hegseth (of all people) can just demand the keys and you hand them right over?

They're labelling Anthropic a supply chain risk, without even the pretense that this is in fact true. They're perfectly content to use the tool _themselves_, but they claim that an unwillingness to sign whatever ToS DoW asks marks the company a traitor that should be blacklisted from the economy.

I thought it was a joke? Like the reviewer is saying, "I didn't finish these books".

If I were a CTO or VP these days I think I'd push for a blanket ban on committing docs/readmes/diagrams etc along with the initial work. Teams can push stuff to a `slop/` folder but don't call it docs.

If you push all that stuff at the same time, it's really easy to get away with this soft lie, "job done". They can claim they thought it was okay and it was just an honest mistake there were problems. They can lie about how much work they really did.

READMEs or diagrams that are plans for the functionality are fine. Docs that describe finished functionality are fine. Slop that dresses up unfinished work as finished work just fucks everything up, and the incentives are misaligned so everyone's doing this.

Well, if you take "review the LLM output" in its most general way, I guess you can class everything under that. But I think it's worth talking about the problem in a bit more detail than that, because someone can easily say "Oh I definitely review the LLM output!" and still be pushing work onto other people.

The fact is that no matter whether we review the LLM output or not, no matter whether we write the code entirely by hand or not, there's always going to be the possibility of errors. So it's not some bright-line thing. If you're relatively lazier and relatively less thoughtful in the way you work, you'll make more errors and more significant errors. You'll look like you're doing the work, but your teammates have to do more to make up for the problems.

Having to work around problems your coworkers introduced is nothing new, but LLMs make it worse in a few ways I think. One is just, that old joke about there being four kinds of people: lazy and stupid, industrious and stupid, smart and lazy, and industrious and smart. It's always been the "industrious and stupid" people that kill you, so LLMs are an obvious problem there.

Second there's what I call the six-fingered hands thing. LLMs make mistakes a human wouldn't, which means the problem won't be in your hypothesis-space when you're debugging.

Third, it's very useful to have unfinished work look unfinished. It lets you know what to expect. If there's voluminous docs and tests and the functionality either doesn't work at all or doesn't even make sense when you think about it, that's going to make you waste time.

Finally, at the most basic level, we expect there to be some sort of plan behind our coworkers' work. We expect that someone's thought about this and that the stuff they're doing is fundamentally going to be responsive to the requirements. If someone's phoning it in with an LLM, problems can stay hidden for a long time.