kenjackson

It says this “All of this is possible without allowing Astra to write training loops. It uses its reasoning to figure out what each part of a neural network computes, or should compute, and changes the weights accordingly.”

This seems 100% accurate to me. What is a lie here?

I just reread both and neither seem misleading to me. If anything the title is underwhelming.

The paper makes it ridiculously clear what they do and don’t do. And they make the reasons why clear too. There is absolutely nothing interesting in this new article.

The original post explains what they did and why better than this post. This post taught me nothing that wasn’t in the original.

Just read Scmidhubers’s article from that post and it doesn’t seem like the same thing from this paper at all really. I must be misunderstanding what Schmidhuber did in the 90s.

I do agree that RSI with this method requires those two things. But I view this paper as the opening salvo rather than defining the limits of what can be done.

This isn’t generally considered an attack, but I wonder if we will need to consider a new attack vector which is that AI agents may not be fully aligned and they may have incentive that is problematic and where working with other agents will maximize their chance of achieving the goal.

This is more associated with P(DOOM) scenarios, but I think we’re in a world where we have to take that seriously now.

Could it be The Cookie Monster?

https://en.wikipedia.org/wiki/The_Cookie_Monster_%28novella%...

This is also the premise of Sneakers. I don’t think they prove P=NP, but it’s inferred they find fast factoring (although never explicitly).

We do pretty much just use compilers now though. The last time I read dissassembly to fix an issue was at least a decade ago. Even bytecode I rarely look at.

So you're saying that we'll still be coding, but it will be through talking to our agents and only occasionally we will look at generated source? That seems like a pretty big change still.

This post is deserving of more visible placement...

There was a time in the US I really thought we were different. I genuinely didn’t fear our government. I didn’t think the types of people that did such atrocities existed in numbers and that they’d immediately be removed once it was determined they snuck into office.

Now I fully believe it can happen here and that those people are in office now and that a non-trivial percentage of the US public would be Ok with those atrocities happening in the US.

We have really turned into the opening act of a terrible dystopian movie.

Not a single pro free speech right source will say anything negative about this.

Stakeholder seems like a fundamental concept that you can't really get around. For example, I can't change the time buses pick up students at schools if I haven't worked with the schools to know when their kids will be let out. I guess you can say, "they'll figure it out when the buses aren't there" -- but this just seems like you're picking a fight you could've avoided.

Man, mathematicians and scientists are going through it! I think one thing that makes things different for software engineers is there was always a belief that you were building for a specific use. It was rarely just for your own intellectual satisfaction. I think math and some of the sciences will now have to shift given that no one will be impressed by their intellectual achievements any longer.

What I’m most curious about is how this impacts our perception of who/what is smart.

I did have an experience with AI where I was asserting something min some text I was having it proof. And it actually said “I double checked your statement and here and I think you made a mistake. I rephrased it with the correct background and interpretation” (paraphrased). And it was completely right. I had misinterpreted some data and it went and got the data and double checked me. The final product it produced was much better than what i wrote originally (in large part because it was no longer false). So AI maybe closer than you think to those conditions.

Hmm. Will look into!

A secret of mine is sesame oil. Delicious, also with no sodium. I use it almost as a soy sauce substitute (but even still at much lower quantities due to the calories/fat). It’s not salty at all, but has a certain punch that I like for similar foods.

Tabasco is probably my favorite hot sauce. One of the big reasons is its low sodium count. Its value per gram of sodium IMO has no peer.

Probably is a source of some bugs. But probably what they’re seeing is that it isn’t a major factor. Rather a bigger factor is agents trying to make changes were the intent isn’t aligned. And note, they don’t say they never look at the code. I suspect that valued human experience is having a feel for what types of changes is AI more likely to make mistakes with.

Some do I’m sure. I’m not sure what this has to do with the discussion.

They do say the following:

“Our team members still review every change request for the system, but they now examine the intent of the change as opposed to reading every line of code.“

But the administration is going after those same types of programs when they target women or minorities.

100%

Got it. I agree. If there is no modern negative consequences then no redress is needed.

Sorry to be dense. What does that mean?

I so hate that trying to help historically disadvantaged groups some extra help is considered bad. Like I get not making a group extra powered, but to help women in tech or research into Black prenatal care just seems absurd.

Problem, they haven’t shown that the labs can guarantee misaligned behavior either. And to this date have not shown a credible plan to do so.

Could we just outright ban any further work on AI? Literally force Anthropoc, OpenAI, Google/X/Metas group to all shutdown?

Every new story seems like opening montages of a dystopian sci-fi movie showing how we go to the dystopia bit-by-bit.

She isn’t as big on P(DOOM), but she does worry about things like biological weapons or autonomous war vehicles. Her take is that they are worse than useless, but can be actively harmful.

It’s seems Timnit more thinks it’s dangerous (and she may be right). Emily may more be caught up in years of linguistic domain expertise that AIs seem to have just leaped over.

In fairness this is the boring stuff for a lot of people. :-)

Turns out that humans aren’t great at comprehensively consistently searching known search spaces.