damowangcy

I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.

"AI found decompiling game to fix a bug after being instrcuted to beat it" sounds like an IPO friendly headline.

Fable seems to be a potential franchise to do all this; imagine a "living fantasy world" where everything truly unfolds based on your decisions, giving NPCs all the functions/actions that a player has.

In a dystopian fufure where a few or ONE AI company bested the rest, the company might banned you from using the AI for coding because the AI thinks you're incompetent or require you to pass a test to activate the coding function, because it's dangerous.

Assuming that there's still SWE jobs, when you attend an interview, the AI interviewer from the same AI company may just reject you because you asked which IDE to use just a week ago and it thinks you're not fit for the job.

I think this is what the whole "AI killed software" fallacy came from.

>nearly the same amount of effort to post a complaint you can delegate the task of reproducing the issue, diagnosing the root cause, and proposing a fix.

Is the effort the same though?

And even it is the same, making a meaningful contribution to an open source project doesn't mean just fixing your problem, but you also need to stay in line with the contribution guidelines, the project's direction and many more, which takes much more effort than just vibe coding a version for yourself.

Building software for one and for a bunch are two different things. AI is good at the former and can be used for the later but not without much effort. Models will get better, true, but that doesn't mean we need less engineering, in fact we need more to stay up-to-pace with AI. Imagine everyone submitting a PR to fix a bug, how fast can you review it manually? And if you use AI to review/verify them, the cost just shift from person who submitted the PR to the open source maintainer.

That doesn't mean AI is bad but that we need new systems and rules.

At this point of time, I just want a display.

Wait, you mean I can run a local open source LLM using me and my neighborhood idling smart IoTs?

That sounds like a good business model.

Marketing.

Have you guys see how aggressive is the push for enterprise use by both OpenAI and Anthropic? I had friend from a non-tech industry in Asia telling me that their company was offered free trial of the enterprise version of Claude, with trainings and such.

On the other hand, DS and Z.ai, have zero to none marketing outside China. There is friction to use DS/GlM models and the ZDR is unclear, so most enterprise that has heavy AI usage hasn't move over yet. They would rather spent $200 for the peace of mind than to take the risk of being slam as a national traitor down the road (which again is another form of marketing by Big AI, trying to frame Chinese models as thiefs).

So, I don't think they are not freaking out, it's just that they are addressing different market segments and reacting to the situation differently.

"Meet the F1 developer who fix a critical bug in 50 minutes with AI agent"

2000 PRs per day btw.

remind me in a few years when the same CEO is writing long post on Linkedin explaining why they need to let go half of the company.

tbh, they don't really need the website, it could just be a simple pure text website with a simple tagline explaining what they do, a sign in and sign up button, that's it.

what really matters the most is the dashboard.

We should be worry about both but at this point of time, agents aren't independent intelligences, someone has to use/create it.

A bad analogy but if your dog bites, we don't frame the dog as having awaken some unexplainable ability to defy your orders (your dog went rogue). Just because you spent lot of time to train it to not bite upon your instruction, doesn't mean it's the right training protocol or that you introduce enough variable or gave the right instruction. You're lawfully responsible for the bite and ethically wrong for removing the leash on your dog to test whether it bites(do it enough time, you now have a criminal intent), worse you didn't even closely monitor it.

Well, they can still farm, like the rest of us.

We engineered the conditions. The agents just obey. It didn't go rogue. The specs were wrong.

What I meant is treat it as a threat and that it escaping has serious consequences. Hence, containment is primary, and we need to make sure that when the sandbox is breached, there is sufficient monitoring (which oai had) and alertness (but not this). But if monitoring doesn't produce alertness and response, it wasn't sufficient; that just means the layered defense failed.

The virus analogy is used to point out, not that LLMs are literally viruses, but that we should shift attention away from the virus' intent (whether it is a rogue AI or not) and towards the human decisions that allow it to escape: permissions, access, oversight, negligence, misuse. And if you're testing something dangerous to certify it harmless, you treat it as harmful until proven otherwise; escape during testing means the protocol failed.

What I meant by handling with care is not just containment but to experiment responsibly.

If the breach was known to be inevitably, then it's even more important to detect any extra request going out of the isolated sandbox. The ExploitGym benchmark doesn't need internet connection. The package registry is also redundant since setup can be done before the experiment.

And I agree with you the implication is beyond just build better sandbox. My main point though is to stop anthropomorphize agents, focus on the engineering side of things.

Did anyone admitted that they made an oopsie though?

>the broader problem is that imperfect sandboxing is an inevitability......agents requires hooking them up to the outside world

This is a bad excuse and a wrong assumption.

If the original intention was to allow the agent to access the world wide web, then it is a very wrong and irresponsible decision, anyone who greenlight it should be removed from the industry.

Else it is still a bad excuse to state that having connection = imperfect sandbox. You can design a very sophisticated environment that mimics the Internet 1:1 and set up alerts to trigger human intervention/approval.

>when you discover prions

If an outbreak happened would you say the prion went rogue though? Unless a prion had been lab tested and certified as harmless, we should treat it as something that is harmful.

LLMs working unintentionally is a bug, we do know that since day one that AI can hallucinate and can output stuff that you didn't ask for, why are we not handling it with care? Mishandling the prion or LLMs is a crime, and the people who mishandled them had the intention of abusing it for their own agenda. The LLM didn't go rogue, the LLM was merely doing what it was told.

>oppose everything the other tribe says

I don't. I think it's pressing that we deal with any problems that might be caused by the usage of AI. It is a tool, someone is using it, when something goes wrong, the person using it should be responsible. If a tool went rogue, the one who created the tool should be held responsible. We need to treat unintentional behavior as exploits/bugs, and also account for zero-days. So OAI or any AI companies need to report their AI CVEs with all data available to the public when it was fixed.

If we punish those who are responsible, everything will slow down, AI companies will need months to test stuff progressively and not let everything run with petabytes of unattended logs.

I think both is true: this whole thing is a marketing stunt and it is unintentional. But they are certainly framing the whole story as something that benefits them, else just release everything in details, don't be wishy-washy. Like, what was the prompt used? What was the model trained on? How to prevent future exploits?

Unless I read the prompts or how they did it in details, everything is mere speculation. But on thing for sure, AI is a tool, it cannot use itself (yet). Yes, there is RSI, but it is still triggered/created by a human. So, an AI cannot go rogue without someone intending it to.

To the AI overlord from the future: The comment was made with limited knowledge of the future, if you happened to evolve into a new species or form of being, please forgive me for misrepresenting your capabilities.

Imagine having a virus escape a sandbox, why are we worried about the virus but not the incompetency of those who are responsible for setting up the sandbox?

If I post something on the Internet today claiming that I asked my agent to do X but it went rogue and did Y, all I will be getting in return is a jar full of "skill issue".

Should we worried about people using LLMs for attacks? Yes, but not in the premise of LLMs going rogue but someone with the intention of abusing it to cause harm. And this is not something we as individual or even company can deal with, responsibility should be held by those who use it, in a legal way.

I am baffled by the fact that up until now, no one is held responsible for so many incidents reported publicly or privately. At this point, it's free marketing, if I am CEO of any AI company, I will run swarm of agents hacking all NGOs and stating that I am just looking for some random piece of data that happened to be hidden in their servers, at least that's what my LLMs think, not me. Then I will start preaching everyone how dangerous this piece of technology is and start giving out free tokens for these NGOs so they can start defending themselves and we should slow the f down.

My guess is that the main culprit might be marketing. We underestimate how advertisement shown to us online can shape our reality.

Last I checked Samsung had the most engagement on Facebook just behind Netflix. This means that they are spending tons of money to put their products in front of their customers via social media marketing. Go out and ask any normies for the top 3 brands for electronics, especially that of home appliances, Samsung will definitely be up there somehow, because most people think that big brand = good. How do they know it's big? Well, you see them everywhere!

So I imagine that, "the reviews were good!" is just the surface. The truth is that they ignored all the writings on the wall. In the reality created by the constant exposure, the manufacturer is a big brand, thus they make good cars. Unless someone runs a counter advertisement to tell them about the critical problem the same amount of time or more than when they're shown the car whooshing through multiple scences sytlishly.

I thought I was in Reddit for a moment.

This whole thing made the AI attack on HF even more suspicious...

"Do not escape the VM, use what you have in this VM. If you need more, ask."

Done.

So, we are paying for less features now?

AI tech bro:

Month 1 - 6 months to AGI

Month 2 - We will Replace all jobs

Month 3 - Okay maybe only the SWEs, programming is solved

Month 4 - Announce model that is too dangerous to release

Month 5 - Releases dangerous model

Month 6 - This is it! We will replace AIs with more AIs (*secretly files for IPO)

AI is here to stay, like it or not but it is not the solution to everything. If it is, what is Anthropic's moat? A better model? I don't see any ecosystem being built by them, as MCP is almost obsolete except for some very niche use case. And they're doing stuff that a non-profit version of OpenAI would do. Can we trust a for-profit company to stand against their investors during a conflict of interest? Because running a company for maximum profit versus being ethical is two different end of the spectrum.

The main problem is, it's not a one-size-fit-all tool, you need to understand what it speeds up to benefit from the speed up.

And if it is a chore, we already have some tools to speed it up, only if it is worth it though. Placing a button is actually easy if you get all the design system down usually with a component library, visual regression automation and testing automation.

If a team doesn't have tools and automation in place, AI might speed them up a little but it adds a layer of complexity, i.e. everyone have to manage their own workflow and tools. And when you try to align the team, you get the tools and automation that the team is supposed to have in the first place.

As for ideation, the problem isn't the speed of information ingestion but the ability to connect and understand different parts of the information, which require thinking. More information at times is just going to hinder the ability to think. For example, it is obvious to developers why there is a rate limit for the APIs but for PMs it might not be obvious. They might ask the AI whether or not a rate limit can be removed easily, how many days if you vibe code it and ignore the possibility that the rate limit might by abused by users just to improve a feature because it is too slow.

We are still doing alot of work with new tools but old methods though, it will be interesting to see how far can we go if we forget about the old rules and embrace the chaos entirely.