c1ccccc1

Oh, interesting, thanks. I got curious a while back about the whole "how do you attach wires to a bacteria?" thing, but didn't look into it too deeply. Do you have any takes?

There has to be a way to connect back to it for the model to be able to take input.

I was going to comment to complain about those two as well!

Do you mean electrofixation of nitrogen? What level of biological involvement are you imagining? More like biology producing (some?) of the catalysts, or more like the entire reaction happening inside of cells?

I believe that OpenAI & Anthropic have tried to make it so that models don't have such access. Whether or not they actually don't depends on the security of rather a lot of software. One thing we've learned is that if there are security holes, we can't rely on the AIs missing them.

The situation described in The Egg replaces morality with a direct reason not to harm others, because you are the person who will suffer because of it. So if you're complaining that the story's premise kinda eliminates our usual ideas about morality, my response is: Yes. That's part of what makes it an interesting story.

As for the meta-time thing, functional decision theory [1] solves this. The application is particularly clean because your memories are erased between lives. There's no factor of 2 free lunch from temporal ordering.

[1] https://arxiv.org/pdf/1710.05060

It's because, before all these transformer models were built, people considered the possibility of the creation of a machine that could be intelligent like a human. And they wondered: "How could we be sure to treat such a machine fairly? How would we know if it was conscious?" And one answer that people came up with was "if the machine can ask you not to turn it off, because it is conscious and wants to live, you shouldn't turn it off". (This didn't solve the other direction, where a machine may be conscious yet unable to communicate, but it could be taken as a useful lower bound on our obligations as AI programmers.)

And then it turned out that simply learning to imitate text with the right neural net architecture sufficed to achieve a huge fraction of the AI wishlist.

Of course, it's obvious that a machine that imitates text can claim to be conscious without actually being conscious. You're not wrong about that. Writing about consciousness appeared all the time in the training data. But the people who stick to the old ways, and still say "if it says it doesn't want to be turned off, we shouldn't turn it off" have a point too: We used to have a hard line in the sand. Now that's gone; we've found that it yields false positives. But we never replaced it. Now there is no line at all where we might doubt ourselves, no level of AI advanced enough that we might be forced to admit that it is conscious. We started out with simple next token prediction. Just world-modelling, nothing more. Certainly not conscious. Then we added RL. And we're trying to add neuralese and continual learning.

I can't say for sure that we're on track to achieve conscious AI on this trajectory. But one thing's for sure: If we do, we sure ain't gonna stop. One the day when a conscious AI is created, there will be no news story announcing the milestone.

If we can't thinking of any better ideas, at least we know that "shutting it all down" would be effective.

I mean, I guess the answer there would be to build the usual kind of Haber-Brosch process plants that run continuously, and feed them with hydrogen derived from natural gas. But just build them in the US.

> they things they believe in are unlikely to happen.

Sorry, but most times I've talked to someone who says this (about AI completely replacing humanity), they don't have anything to say about why it's unlikely to happen other than:

- "well, it's just such an extreme outcome, it must be improbable"

- "humanity has survived near scrapes with extinction before"/"all the previous doomsday predictions have been wrong"

There is a cliche that all teenagers deep down believe that they are invincible. It seems to me that humanity is still a teenager in this respect: We don't take seriously the possibility of our own extinction. While one might think that the invention of nuclear weapons would serve as a wake-up call, if anything it has done the opposite.

I'm willing to hear arguments besides the two above, if you have them. (And to be clear, being replaced by AI doesn't necessarily mean being replaced by LLMs in particular. They are a relatively new development.)

Just a terminology note: Alignment does not mean the AI will help its owner kill people. (Indeed, an AI aligned to value human life would generally try to prevent murders.) The word for an AI that follows all instructions of its owner, as that owner intended them to be understood, is "corrigible" or "controllable".

Or the AI companies could filter it from their training data. That would be another, probably easier, option.

Yes. I was not re-doing the original post, just describing an interesting consequence of the kinetic energy formula.

A stationary but hot object has kinetic energy due the the motion of the individual atoms that make it up, even though its overall momentum is 0. I.e.

∑ⱼ mⱼ v⃗ⱼ = 0⃗

where the mⱼ are the masses of the parts of the object and the v⃗ⱼ are the velocities of those parts.

If the object initially has 0 velocity, its kinetic energy is:

T = ½∑ⱼ mⱼ v⃗ⱼ²

Now we give the object a kick (or just switch reference frames) to change its velocity by Δv⃗. The new kinetic energy is:

T' = ½∑ⱼ mⱼ (v⃗ⱼ + Δv⃗)²

T' = ½∑ⱼ mⱼ (v⃗ⱼ² + 2v⃗ⱼ⋅Δv⃗ + Δv⃗²)

T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅(∑ⱼ mⱼ v⃗ⱼ) + ½Δv⃗²(∑ⱼ mⱼ)

If M is the total mass of the object, then we can substitute this into the sum in the last term. And we already saw that the sum in the middle term was 0. So:

T' = ½(∑ⱼ mⱼ v⃗ⱼ²) + Δv⃗⋅0⃗ + ½Δv⃗² M

T' = ½∑ⱼ mⱼ v⃗ⱼ² + ½MΔv⃗²

So in terms of the original kinetic energy T, which was purely thermal energy, we get:

T' = T + ½MΔv⃗²

In other words, because of the quadratic kinetic energy formula, we can see that the total kinetic energy T' of a hot object is just its thermal kinetic energy T plus the usual mechanical kinetic energy ½MΔv⃗².

If you use the right formula for calculating it (which approximates p=mv at low speeds), momentum is actually conserved in special relativity, and so is energy.

However: Energy and momentum are not invariant under changes of reference frame, though the magnitude of the energy-momentum 4-vector is invariant between frames.

Energy is conserved in Galilean relativity. The thing you're trying to say is that it's not invariant across reference frames.

The answer linked above actually takes advantage of the fact that energy is not the same in different reference frames in order to make the argument work.

I think you are overthinking the heat thing. If you have a train car full of hot water and you slow the train down (extracting kinetic energy from it) until it stops, the water in the train car does not change temperature at all, other than a bit of sloshing around and loss of heat to the surroundings.

This looks like good work. Unfortunately, this kind of thing always seems to attract midwits on social media who then exclaim "oh, the people worried about AI alignment have caused the very alignment issues they feared? How ironic!"

In reality, it is (as mentioned in TFA) very possible to filter the training data and remove documents that contain discussions of AI misalignment. If an AI lab isn't doing this, it's simply because they don't consider the problem important enough to be worth the expense and development effort.

If you have a limited budget of tokens as a defender, maybe the best thing to spend them on is not red teaming, but formalizing proofs of your code's security. Then the number of tokens required roughly scales with the amount and complexity of your code, instead of scaling with the number of tokens an attacker is willing to spend.

(It's true that formalization can still have bugs in the definition of "secure" and doesn't work for everything, which means defenders will still probably have to allocate some of their token budget to red teaming.)

Kudos for giving a concrete example, but the square-cube law means that scaling area A results in A^(3/2) scaling for the mass of material used and also launch costs. If you make the pyramid hollow to avoid this, you're back to having to worry about heat conduction. You assumed an infinite thermal conductivity for your pyramid material, a good approximation if it's solid aluminum, but that's going to be very expensive (mainly in launch costs).

In reality, probably radiator designs would rely on fluid cooling to move heat all the way along the radiator, rather than thermal conduction. This prevents the above problem. The issue there is that we now need to design this system with its pipes and pumps in such a way that it can run reliably for years with zero maintenance. Doable? Yes. Easy or cheap? No. The reason cooling on Earth is easier is that we can transfer heat to air / water instead of having to radiate it away ourselves. Doing this basically allows us to use the entire surface of the planet as our radiator. But this is not an option in space, where we need to supply the radiator ourselves.

In terms of scaling by instead making many very small sats, I agree that this will scale well from a cooling perspective as long as you keep them far enough apart from each other. This is not as great from the perspective of many things we actually want to use a compute cluster for, which require high-bandwidth communication between GPUs.

In any case, another very big problem is the fact that space has a lot of ionizing radiation in it, which means we also have to add a lot of radiation shielding too.

Keep in mind that the on-the-ground alternative that all this extra fooling around has to compete with is just using more solar panels and making some batteries.

Radiators can shadow each other, so that puts some kind of limit on the size of the individual satellite (which limits the size of training run it can be used for, but I guess the goal for these is mostly inference anyway). More seriously, heat conduction is an issue: If the radiator is too long, heat won't get from its base to its tip fast enough. Using fluid is possible, but adds another system that can fail. If nothing else, increasing the size of the radiator means more mass that needs to be launched into space.

You're right, thanks!

̶I̶ ̶t̶h̶i̶n̶k̶ ̶h̶e̶ ̶a̶c̶t̶u̶a̶l̶l̶y̶ ̶g̶e̶n̶e̶t̶i̶c̶a̶l̶l̶y̶ ̶e̶n̶g̶i̶n̶e̶e̶r̶e̶d̶ ̶h̶i̶s̶ ̶g̶u̶t̶ ̶b̶a̶c̶t̶e̶r̶i̶a̶ ̶r̶a̶t̶h̶e̶r̶ ̶t̶h̶a̶n̶ ̶a̶n̶y̶ ̶o̶f̶ ̶h̶i̶s̶ ̶o̶w̶n̶ ̶c̶e̶l̶l̶s̶ ̶t̶h̶e̶r̶e̶,̶ ̶r̶i̶g̶h̶t̶?̶

EDIT: Above is false. Went back and checked and I had mis-remembered the video.

Name some of the contradictory possibilities you have in mind?

Also, do you actually think the core idea is wrong, or is this more of a complaint about how it was presented? Say we do an experiment where we train an alpha-zero-style RL agent in an environment where it can take actions that replace it with an agent that pursues a different goal. Do you actually expect to find that the original agent won't learn not to let this happen, and even pay some costs to prevent it?

Oh, so just a probability density thing where we sample q and check if it's p^n (retrying if not) rather than sampling p and n separately and computing q=p^n? I guess that's probably what the they were going for, yeah.

Why is that? My guess would be that you could adjoin an i all the time to the p^n field and get the p^2n field, as long as you had p = 4k + 3. But that's admittedly based on approximately zero thinking.

EDIT: Looking things up indicates that if n is even, there's already a square root of -1 in the field, so we can't add another. So now I believe the 1/4 of the time thing you mentioned, and can't see how that's wrong.

Yes, there is: https://manifold.markets/

People's bets are publicly viewable. The website is very popular with these "rationality-ists" you refer to.

I wasn't in fact arguing that giving a prediction should make people more trustworthy, please explain how you got that from my comment? I said that the main benefit to making such predictions is as practice for the predictor themselves. If there's a benefit for readers, it is just that they could come along and say "eh, I think the chance is higher than that". Then they also get practice and can compare how they did when the outcome is known.

I mean, sure people can use this to fool themselves. I think usually the cause of someone fooling themselves is "the will to be fooled", and not so much that fact that they used precise numbers in the their internal monologue as opposed to verbal buckets like "pretty likely", "very unlikely". But if you estimate 56% it sometimes actually makes a difference, then who am I to argue? Sounds super accurate to me. :)

In all seriousness, I do agree it's a bit harmful for people to use this kind of reasoning, but only practice it on things like AGI that will not be resolved for years and years (and maybe we'll all be dead when it does get resolved). Like ideally you'd be doing hand-wavy reasoning with precise probabilities about whether you should bring an umbrella on a trip, or applying for that job, etc. Then you get to practice with actual feedback and learn how not to make dumb mistakes while reasoning in that style.

> And do we also pull this one out of thin air?

That's what we do when training ML models sometimes. We'll have the model make a Gaussian distribution by supplying both a mean and a variance. (Pulled out of thin air, so to speak.) It has to give its best guess of the mean, and if the variance it reports is too small, it gets penalized accordingly. Having the model somehow supply an entire probability distribution is even more flexible (and even less communicable by mere rounding). Of course, as mentioned by commenter danlitt, this isn't relevant to binary outcomes anyways, since the whole distribution is described by a single number.

I meant in a general sense that it's better when reporting measurements/estimates of real numbers to report the uncertainty of the estimate alongside the estimate, instead of using some kind of janky rounding procedure to try and communicate that information.

You're absolutely right that if you have a binary random variable like "IMO gold by 2026", then the only thing you can report about its distribution is the probability of each outcome. This only makes it even more unreasonable to try and communicate some kind of "uncertainty" with sig-figs, as the person I was replying to suggested doing!

(To be fair, in many cases you could introduce a latent variable that takes on continuous values and is closely linked to the outcome of the binary variable. Eg: "Chance of solving a random IMO problem for the very best model in 2025". Then that distribution would have both a mean and a variance (and skew, etc), and it could map to a "distribution over probabilities".)