ankithg

The privacy concerns here are valid, but there's a second axis that gets less attention: an agent with its own browser and access to your accounts takes actions, it doesn't just observe. The failure mode isn't only "Meta sees your data", it's "the agent misunderstands you and does something you can't take back" - books the wrong flight, sends the wrong message, buys the wrong stroller. And for all these personal-agent launches, nobody ships public evidence of how the thing behaves under ambiguous or adversarial instruction. The evals that exist are vendor-run and pre-launch. What these systems actually do in the wild, with real users pushing on them in ways the vendor didn't script, is basically unmeasured.

One thing the doom debate tends to skip: the failures that hurt people right now are boring, not exotic. I spend my time probing AI systems that are already deployed and talking to real users, and the actual failure modes are mundane - a system that reassures when it should escalate, follows its script while the situation changes, sounds confident the whole way through. No superintelligence required, just an ordinary model given authority and no tripwires. If the conversation centered on the systems already in production instead of hypothetical future ones, a lot of the disagreement about timelines would matter less. The dangerous stuff is already here; it's just unglamorous.