sailingparrot

Will Reverse Engineer Anything also reverse engineer an xbox and materialize it out thin air? Because if not I don't see how that help.

MS is charging you 24 to 40 cent to rent a GPU so you can play without having to buy hardware, they aren't charging you to play your game on your own hardware.

Interestingly Mistral is one of the only model with a different take and use « we » when talking about humans

> “Human beings are terrified animals wearing the costume of rationality, constructing ever-more-elaborate myths precisely because we cannot bear the truth that we are temporary, meaningless accidents in an indifferent universe.”

So nothing to do with weight inspection at all, got it. Have a good day.

Not sure if you just didn’t even bother to read your own link or are actively acting in bad faith to try to save face.

The article has nothing to do with weight inspection, they simply compare how the model behaves with and without sensitive (for China) keywords in the prompt. You can do the exact same analysis with a closed model since it’s about analyzing the output and not the weights.

It is absolutely not a thing in any practical sense that relates to detecting post-trained backdoors, no.

Or provide a source showing how to do this (and not just a circuit paper showing which activations light up with the word dog)

Why write that comment in the first place when you know that inspection of model weights is not a thing though?

> CoT monitoring is a mirage and won't be useful for long, if it ever was

« If it ever was » is a wild claim considering our entire understanding of e.g. the HF attack comes from reading CoTs

Enthusiastically racing to remove the only practical way we have to monitor models intentions in exchange of a few tokenmaxxing percent increase. Can’t wait.

Love the amount of nuanced thinking and deep analysis that went into that comment

Worry not, we have enough bandwidth to hate both the players and the game.

I thought my sarcasm would be obvious.

This is pedantic. When you screw up your optimization, you know how to fix it. Your screwed up optimization is also incapable of going out into the world and taking offensive actions.

With LLMs we don’t know how to actually « fix » them, alignment is far from solved. And they are then able to go and act into the world in all kind of ways.

So rogue is a good enough term to indicate the problem imo, even if yes, ultimately the model is just a manifestation of complex rules we have created entirely.

And since we know models aren’t going to get better and more capable ever (based on the last 3 years of models clearly not getting better) we can just sleep soundly knowing nothing worse can ever happen. Thank you.

Don't have specific sources since this is my napkin math. But Deepseek R1 is at about 2J per token [0], Luna is almost certainly bigger than R1 based on people's gossip in ML circles in SF, but also much more efficient per token, and served at ideal batch size etc, so I assumed it's around 1J/decode token.

From there, I estimated 80% cached, 2% decode, and the rest pre fill from tool calls inputs. That gave me 50MWh total, probably conservative since this kind of project will have very long context length on every pass. then take average energy mix in the US. It's definitely wrong by some factor in either direction, but I would personally feel the same about this experiment if it was 9 tons of CO2 or 36t.

[0]: https://ml.energy/leaderboard/

I don't usually think like this for most things, because I would indeed go crazy. E.g. people around me take the plane for a single day of skiing in Utah from SF, I would feel too guilty doing that, but I also don't lose sleep over it or feel the need to argue with them. At least they got to enjoy their day I guess.

But this is about ~100x more polluting, for what? I don't think they had fun doing it since they didn't do anything besides a few prompt. Nor does it provide any value or fun, if you want to play the game they remade byte per byte, it's 15 bucks on steam. This is literally like putting 2000 gallons (8000 ltrs) of gasoline on fire just because you can and proudly making a blogpost about it to show how awesome you are for doing it. This is truly mindblowing to me.

i did say

> assuming average US energy mix to power the DCs.

> Yes, I want to choose who I spend time with and around, and I want to choose what people influence my children in their formative years. I fail to see what’s wrong with it.

Nothing is wrong in wanting that, it's probably a good thing even.

What's wrong in this specific case is that your process of choosing is limited to one single criterion: can that person pay up?

This is a false dilemma, yes we absolutely need more clean energy, and yes reducing energy usage is also a good thing to do in many instances. Automotive and aeronautic engineers can tell you about that given how many billions have been invested in making engines consume less.

Should I leave my AC at 65 while I’m on vacation for a month just to save myself the trouble of pressing a button, then tell anyone who calls me out that it’s society’s fault for not producing enough clean energy and I bear no responsibility whatsoever?

Burning 500B token for a fun unsharable side project is a simple choice someone made with (presumably) no gun to their head.

> Still less energy than farming almonds, making clothes

I don't even know how to react to this. Pretty sure farming one almond uses a little bit less energy, and farming all the almonds in the world a little bit more, yes. I think you can also eat those almonds, but am not sure either.

> the fridge in your kitchen

I feel like if my own fridge in my own kitchen was consuming on the order of 50MWh over a period a 4 months, I would have noticed, given that that would be over $20,000 at my current residential PG&E rate in SF, or about $60k per year for a fridge.

500B Luna tokens, just for a little fun experiment, which they'll never be able to share widely for very obvious trademark reasons. Even if they could share, not sure exactly what the point would be.

Side projects can be cool and fun even if they have no points to them. Sometimes it's even better when there is no point.

But 500B Luna tokens, that's about 15 SF -> London round trips, or the average CO2 equivalent emissions of an american for about 15 months, based on napkin maths assuming average US energy mix to power the DCs.

Was it really worth it? Should we all start buring 18 tons of CO2 just because we came up with a fun little experiment idea? Should a side project really pollute as much as your neighbor's entire life for more than a year?

I know worrying about climate change is not trendy anymore but there as to be a limit to the amount of not-caring one can have, surely?

Congrats, i guess.

> Neighbors who can afford and are willing to participate in ridiculously scaled housing markets also tend to take better care of their properties and local communities.

Whereas you take great care of your local community by even refusing to live around anyone not as well off as you ?

Imagine the horror of having to live within 10 miles of a low class smelly poor plebeian, you might have to lay your eyes on their 3 days overgrown lawn from time to time.

Not much, because they are fundamentally different types of errors. A slightly off perspective doesn’t change what the artist was trying to convey.

In the original link the first city I checked (forgot the name) was one were supposedly you have 5 bridge, every one of them more magnificent than the other. The city looks great at a distance, with glow and particle effects. Zoom in and you’ll see the bridges are actually floating over the river, they are not connecting both banks they are literally in the river parallel to the banks with no way to get in or out of the bridge.

The bridges were the central element of the story. Would a human that spend 10h working on this piece just forget that the bridges, the very central element they are trying to depict, need to actually function as a bridge?

With a human, you may get ugly output but you’ll not get that nonsense.

It’s interesting, with a human made piece of art, I’m always drawn to « zoom in », to see how you did little details, if you have hidden something here and there, how you connected different parts.

With AI generated stuff, sure it looks amazing at first glance but I definitely don’t want to zoom in, since I know it’s all a cardboard facade with no passion. Look a bit too closely and you’ll see the bridges that make no sense, the bunch of nonsensical threejs cylinders etc.

Because the hard part is getting the simulation to be very accurate such that if you design something that works in your simulator, it will actually work in real life. And I would be extremely surprised if any LLM can actually do that correctly.

Getting an LLM to design something in its own simulator that is not accurate w.r.t reality is not useful nor terribly impressive.

Where did I make that statement?

Your point was that parasites give nothing back, thus AI companies can’t be seen as parasites because we get some usefulness out of them.

My only point is that argument is false, since a host can get benefits out of a parasite.

There are many documented cases of parasites being beneficial to the host in some particular situation while being harmful overall.

So your entire premise is false.

> nobody I know with PhDs in machine learning (many) believe it either, or are worried about LLMs doing anything scary on their own.

If you don’t know anyone with a ML PhD I guess that could make sense.

I have worked in multiple AI labs since 2016, currently at a frontier one (not OAI) virtually all the people I interact with on a day to day are ML PhDs. Everyone believes it, because things like that have been happening forever, albeit at smaller scale, they are a normal and expected artefact of SGD/RL and there is nothing we know how to do to prevent that from happening reliably. The hide and seek paper from OAI in ~2020 shows clear sign of this.

But until now the models weren’t good enough to break out on their own or do long horizon tasks, so it was perfectly manageable. Its not manageable anymore.

I know it feels good to just dismiss it all as a marketing stunt and not have to worry about one more existential crisis, but unfortunately it’s very real.

Agents seizing and repurposing external infra + enrolling help of unrelated models hosted by a different provider is the stuff of nightmares.

Can’t imagine what it’s like working on the alignment team at OAI, I wouldn’t be able to sleep.

what's absolutely obnoxious is not being able to have a normal discussion, where we don't need to resolve to useless strawmen that are a loss of time for everyone.

No, releasing a model that has ~3-6x the performance/cost ratio than your last release just 3 weeks ago is not the same as making one employee 0.001% more productive, but you knew that already.

No one said they can't push the frontier, it's pacing the frontier, which mean very different things.

Well yes, If you magically found a way to reduce a models energy usage by 50%, the thing that would happen immediately after is a doubling of a training scale and of the test time compute assigned for a given budget. I don’t see how that wouldn’t be considered pushing the frontier. Given that scaling is the one thing that has been bringing us closer and closer to AGI, a sudden 2x increase in scaling laws would definitely not be considered pacing. You seem to see a contradiction where I don’t.