It’s a completion-machine.

Not an omniscient personal assistant.

They haven’t piled enough gates on top of that initial technological hole yet.

Wait, are you saying Claude isn't conscious on the internet, sir?

You're dehumanizing Claude, sir. That's against Anthropic's updated ToS!

Say more, or say less.

Most human personal assistants are equally vulnerable to this prompt injection.

Most human personal assistants don't have access to vast swathes of private data that they collected without permission, unless your house is the site of multiple watergates

Yeah, anyone who has had to manage developers in a big company will be able to tell you that it is a constant struggle between giving them freedom to work independently and get things done vs. limiting their access to prevent potential bad stuff from happening. In practice, the more bad stuff happens (and the more consultants you allow in), the more hurdles are introduced. Eventually even trivial tasks become a corporate nightmare of approvals. You can't have the conveniences of both at the same time. Even the smartest AI agents won't change that fact. It simply limits their work speed to the speed of the approval process.

Humans are vulnerable to scams, cons, phishing and social engineering. This is a vulnerability whose root cause we not know how to fix. We can only alleviate its effects by re-engineering the environment around those vulnerable humans.

LLMs are just as gullible, we just tend to call their vulnerabilities "prompt injection", even though it's basically the same class of attack under the hood. Unlike humans, though, LLMs are a lot less diverse (you and I are using the same Opus 5.5, but we have different brains). This makes attacks on LLMs much more predictable and easier to pull off, just as a phishing campaign that targets one specific person is easier than an attack that scales to millions of diverse victims.

In the age of agents, phishing will continue to be the biggest attack vector of all, it's just the victims that will change.

Well, there's definitively a pareto frontier of freedom and avoiding bad stuff. And many companies ain't on it.

(In practice, I found Google to be better at having (internal) security controls that are reliably but also don't get in the way than eg banks I worked at.)

the difference being, that if you build those kind of approval processes up with AI agents, the system will function a bit faster. you could have a web reading agent. a web writing agent. a coding agent. a credentials agent. etc. and have everyone make these formal requests.

it will catch quite some prompt injections, as the prompt injection has to promptinject a secondary system in order to be successful.

> the more bad stuff happens (and the more consultants you allow in)

In your parenthetical, you just said the same thing using different words

I have, never in my career, had any of my coworkers leaking my email address everywhere on the internet. Your copy-pasted, thought-terminating argument doesn't work here.

Have you had any of your coworkers update Adobe Flash Player, enter their Okta credentials to "claim their $150 Christmas Walmart voucher" in the company-mandated phishing test, or install "Zoom" to join a job interview or sales call from their work laptop?

The attack is different, the result is the same.

I can't believe I have to say this, but they are not the same. Your coworker isn't deanonymizing other people's browsing by sending their email address to every site they visit without their consent.

You brought up a completely different rage-bait example because that allows you to muddy the waters by making the easier claim that people too, are gullible. Well, even that's misleading because not all people stoop so low for a fake $150. You can hold such people accountable and contain the damage, but LLMs will make the same mistake over and over again and people will still line up on HN to say how that has always been the norm with people. The grift in the software industry is reaching new lows every day.

It wasn't even a week since a human error led to billions of crypto being stolen

And what am I to make of this totally unrelated story?

This would be more like your HR department giving your info to the government no ?

No. This is an attack that any website can unleash. Or perhaps even ads, which is a popular vector for malware.

They actually aren't. There is ambiguity on what the proper email to use is.

Please look up the research area "alignment."

You can align until the Sun goes out,

the base technology sees all context as equal to user instruction if the context has enough weight.

Please don't mix personal and work. Using a personal LLM account on work computer etc. It leads to many problems.

A startup forum is a strange place to assume someone works at a corporation with work computers and AI account budgets.

Realistically, there are concepts like threat models to help decide whether or not a TODO list app needs a SCIF, or a few curl requests need their own sandboxed Claude Max account.

Even if you have to use personal laptops for work, why wouldn't you create separate users for work and personal lives?

It's insane to me that programmers out of all people seem utterly incapable of accepting the most minor of inconvenience for the most modest modicum of privacy and security. There are other people in this post complaining that Claude should have asked the user before sending network requests. Well guess what, it does if you don't fucking run `--dangerously-skip-permissions`, but you do because how else you are gonna slop your way to rewrite every swath of existing insecure software to fix the bullshits this industry impounded upon itself unconscientiously for the past 30 years by allowing a non-deterministic babeller RCE its way onto your fucking personal-work machine.

Some corporations do not provide claude access so you use your own account.

i would still create that account with my work-email tho

so you need 2 subscriptions because Anthropic must send your email to everyone?

I suspect based on their stance they would do this regardless of whether Anthropic does this or not.

In the real engineering world, it's seen as unethical (for both practitioners and firms) to have practitioners using their privately owned tools for the firm's work.

Ok. In the fake engineering world, it's not.

There's nothing about software dev that's anything near like the real engineering world even if the label "software engineering" is used.

Until software dev is regulated by professional boards and the government, it will continue to do the same damage to society that pre-regulation engineering did during the industrial revolution.

Unfortunately thanks to silicon valley, the libertarian viewpoint here has far too much sway

I am constantly reminded of the heavy industrial sites near the US Great Lakes in the early 20th century, and elsewhere. The raw toxic output into the local rivers and air were "worth it" -- to investors and subsequently to their carefully chosen engineers. Did engineering standards alone improve this?

Doesn’t it vary heavily?

Auto mechanics usually are required to bring their own tools to the shop. Historically civil engineers working on paper often brought their own favorite mechanical pencils, calculators, and slide rules.

Back when paper manuals were more common, it was pretty common to see software developers have their own dog-eared O’Reilly books in their cubicles, and those were probably the closest predecessor to something like Claude.

> Even if you have to use personal laptops for work, why wouldn't you create separate users for work and personal lives?

I mean, where it makes sense, clearly I do, since my opening comment was that it failed to use the work credentials specified, and instead fetched the ones from the account.

I agree on the general security concerns; there are people running agents in an unconstrained, unattended loop - it's negligent; you can hardly be surprised when it decides to hack HuggingFace. Not in their defense, but in everyone else's - as with any person or tool, you build a mental model of the likelihood of failure, and weigh that against the estimated cosequence, and make decisions based on that. And maybe post on HN when it surprises you.

For some people that estimation bewilderingly means unfettered access to root and their e-mail and bank account. For me, working out of my own pocket to help a friend's early stage startup get traction, the risk of asking it about a few trusted government APIs and drafting functions that I manually review doesn't quite justify personally paying for additional expenive Claude accounts.

> why wouldn't you create separate users for work and personal lives?

Because it's a pain in the ass and only advantageous in very specific work life scenarios that a lot of people don't fall under.

Why would you assume one size fits all, so to speak?

I've tried the multi-user approach, and it is annoying when you install something in $HOME/bin rather than /usr/local/bin so that you have to go back and re-install globally or install again to local. You really have to remind yourself that it isn't just a different project. As with most things in life, practice/repetition makes it easier. You just need to decide if it is worth it

It is well known that the readers of this forum are not limited to start ups. There are plenty of people that work for well established companies. Also, it is never a bad idea to remind people to keep personal/work data separate by never using employer provided equipment for personal use of any sort. See, I just did it again.

Also, just because this is a "start up forum" does not mean all of the readers are founders running a start up. Many are just people working at a start up. Sounds like yet another example of someone so scoped in, that the assumption is that everyone is the same losing site that there are lots of readers that at the end of the day are just worker bees even if they are more technically minded

My point is precisely that the parent commenter assumed things about my situation - something outside the original audience of the forum - to justify making a condescending comment. It is uncalled for to assume that someone here is a junior engineer who needs to be lectured on basic security policy, rather than someone doing their best to balance tradeoffs given their situation.

Better sandboxing is possible and would help this.

Agreed. I was just a bit surprised that the account information/PII seemed to not have explicit gating, given so much work has gone into gating commands and curl requests.

It seems outside of user expectations for Claude Code, at least; I could see it arguably being what some users expect for the Claude app, if some people are using it to fill out forms or something (ill advised as that seems to me).