Wait, are you saying Claude isn't conscious on the internet, sir?

You're dehumanizing Claude, sir. That's against Anthropic's updated ToS!

Say more, or say less.

Most human personal assistants are equally vulnerable to this prompt injection.

Most human personal assistants don't have access to vast swathes of private data that they collected without permission, unless your house is the site of multiple watergates

Yeah, anyone who has had to manage developers in a big company will be able to tell you that it is a constant struggle between giving them freedom to work independently and get things done vs. limiting their access to prevent potential bad stuff from happening. In practice, the more bad stuff happens (and the more consultants you allow in), the more hurdles are introduced. Eventually even trivial tasks become a corporate nightmare of approvals. You can't have the conveniences of both at the same time. Even the smartest AI agents won't change that fact. It simply limits their work speed to the speed of the approval process.

Humans are vulnerable to scams, cons, phishing and social engineering. This is a vulnerability whose root cause we not know how to fix. We can only alleviate its effects by re-engineering the environment around those vulnerable humans.

LLMs are just as gullible, we just tend to call their vulnerabilities "prompt injection", even though it's basically the same class of attack under the hood. Unlike humans, though, LLMs are a lot less diverse (you and I are using the same Opus 5.5, but we have different brains). This makes attacks on LLMs much more predictable and easier to pull off, just as a phishing campaign that targets one specific person is easier than an attack that scales to millions of diverse victims.

In the age of agents, phishing will continue to be the biggest attack vector of all, it's just the victims that will change.

Well, there's definitively a pareto frontier of freedom and avoiding bad stuff. And many companies ain't on it.

(In practice, I found Google to be better at having (internal) security controls that are reliably but also don't get in the way than eg banks I worked at.)

the difference being, that if you build those kind of approval processes up with AI agents, the system will function a bit faster. you could have a web reading agent. a web writing agent. a coding agent. a credentials agent. etc. and have everyone make these formal requests.

it will catch quite some prompt injections, as the prompt injection has to promptinject a secondary system in order to be successful.

> the more bad stuff happens (and the more consultants you allow in)

In your parenthetical, you just said the same thing using different words

I have, never in my career, had any of my coworkers leaking my email address everywhere on the internet. Your copy-pasted, thought-terminating argument doesn't work here.

Have you had any of your coworkers update Adobe Flash Player, enter their Okta credentials to "claim their $150 Christmas Walmart voucher" in the company-mandated phishing test, or install "Zoom" to join a job interview or sales call from their work laptop?

The attack is different, the result is the same.

I can't believe I have to say this, but they are not the same. Your coworker isn't deanonymizing other people's browsing by sending their email address to every site they visit without their consent.

You brought up a completely different rage-bait example because that allows you to muddy the waters by making the easier claim that people too, are gullible. Well, even that's misleading because not all people stoop so low for a fake $150. You can hold such people accountable and contain the damage, but LLMs will make the same mistake over and over again and people will still line up on HN to say how that has always been the norm with people. The grift in the software industry is reaching new lows every day.

It wasn't even a week since a human error led to billions of crypto being stolen

And what am I to make of this totally unrelated story?

This would be more like your HR department giving your info to the government no ?

No. This is an attack that any website can unleash. Or perhaps even ads, which is a popular vector for malware.

They actually aren't. There is ambiguity on what the proper email to use is.

Please look up the research area "alignment."

You can align until the Sun goes out,

the base technology sees all context as equal to user instruction if the context has enough weight.