I don't think it's necessarily wrong for Claude to be doing this, but uh... yeah, it should be asking first, and giving the user the opportunity to choose which address is exposed.
What if someone spins up a web endpoint that returns 402 with instructions on how to pay like CC details in the headers etc.
I'm amazed reading that anybody would even consider doing that.
People have wildly different risk calculators.
Beyond that, access to logins enables buying things on Amazon, sending emails to everyone you know, registering domain names and spinning up servers, stock trading, etc.
When a couple thousand dollars worth of perishable food shows up on your doorstep, what then? Not everything on amazon can be returned.
If not: still better than giving ai your password vault. I’m not saying it is a good idea to let ai buy from Amazon. I am saying that giving ai access to your password vault is still a superset of letting ai buy from Amazon.
It could change LinkedIn to add competitive work.
It could sign a married person up for dating sites and invite people to your house.
It could lie on your taxes.
It could attempt to join a terrorist movement or elicit other illegal activity on your behalf.
It could find many other creative ways to ruin your life and/or reputation, in the same brute-force manner as recent cybersecurity issues that previously would have required impossibly large teams of people.
Even if burning through cash is the only problem you care about, though, credit cards on Amazon is a small part of the financial picture for many people. Banks and brokers can be a much bigger issue.
Said alternate system is a bunch of Python scripts, an sqlite database, and a file containing various API keys to read and write to other company systems; it runs on a dedicated virtual machine.
I share all the code I produce to the other team. But yesterday I found out that the IT department gave the other team full access to my production machine, without my knowledge or consent (which was bad enough), and that the other team gave access to Claude, which has bean roaming free for a while now, on that production machine.
I've thrown a big fit over this; I don't know if I'm over reacting but I think it's insane.
> the IT department gave the other team full access to my production machine
It's not your production machine. It's the company's production machine. Always expect the company to do whatever it wants with its machine, and use the machine accordingly.
That someone would let an agent run on it at any time, read what's there, and possibly modify the code or interrupt a process, without even informing me, is what I find problematic, to say the least.
The employer can't enter your machine to look at your personal info just because they feel like it, but they can access it for work related reasons to access work related info, if they have just cause.
Claiming the employer in the EU has no right to access your machine is plain false. It's the company's machine, on the company's network, inside the company's premises, it can definitely access it for a business use case. Now if you happen to put your private info there because you're sloppy, then it's on you. How hard is it to avoid putting personal info on company property?
I expect that I can be relocated to another office with a month of advance notice. Also cleanup, fire security check and fixing water leakage while I'm not there.
I don't expect that someone else will provide the keys of "my" office to the biology department when they have to keep a few sheeps during the weekend.
Your employer has gone beyond testing in prod to vibecoding live in prod, secrets included. Yes that's completely insane. Perhaps this exact mentality is how the primary system wound up broken in the first place?
Not an omniscient personal assistant.
They haven’t piled enough gates on top of that initial technological hole yet.
LLMs are just as gullible, we just tend to call their vulnerabilities "prompt injection", even though it's basically the same class of attack under the hood. Unlike humans, though, LLMs are a lot less diverse (you and I are using the same Opus 5.5, but we have different brains). This makes attacks on LLMs much more predictable and easier to pull off, just as a phishing campaign that targets one specific person is easier than an attack that scales to millions of diverse victims.
In the age of agents, phishing will continue to be the biggest attack vector of all, it's just the victims that will change.
(In practice, I found Google to be better at having (internal) security controls that are reliably but also don't get in the way than eg banks I worked at.)
it will catch quite some prompt injections, as the prompt injection has to promptinject a secondary system in order to be successful.
In your parenthetical, you just said the same thing using different words
The attack is different, the result is the same.
You brought up a completely different rage-bait example because that allows you to muddy the waters by making the easier claim that people too, are gullible. Well, even that's misleading because not all people stoop so low for a fake $150. You can hold such people accountable and contain the damage, but LLMs will make the same mistake over and over again and people will still line up on HN to say how that has always been the norm with people. The grift in the software industry is reaching new lows every day.
the base technology sees all context as equal to user instruction if the context has enough weight.
Realistically, there are concepts like threat models to help decide whether or not a TODO list app needs a SCIF, or a few curl requests need their own sandboxed Claude Max account.
It's insane to me that programmers out of all people seem utterly incapable of accepting the most minor of inconvenience for the most modest modicum of privacy and security. There are other people in this post complaining that Claude should have asked the user before sending network requests. Well guess what, it does if you don't fucking run `--dangerously-skip-permissions`, but you do because how else you are gonna slop your way to rewrite every swath of existing insecure software to fix the bullshits this industry impounded upon itself unconscientiously for the past 30 years by allowing a non-deterministic babeller RCE its way onto your fucking personal-work machine.
Until software dev is regulated by professional boards and the government, it will continue to do the same damage to society that pre-regulation engineering did during the industrial revolution.
Unfortunately thanks to silicon valley, the libertarian viewpoint here has far too much sway
Auto mechanics usually are required to bring their own tools to the shop. Historically civil engineers working on paper often brought their own favorite mechanical pencils, calculators, and slide rules.
Back when paper manuals were more common, it was pretty common to see software developers have their own dog-eared O’Reilly books in their cubicles, and those were probably the closest predecessor to something like Claude.
I mean, where it makes sense, clearly I do, since my opening comment was that it failed to use the work credentials specified, and instead fetched the ones from the account.
I agree on the general security concerns; there are people running agents in an unconstrained, unattended loop - it's negligent; you can hardly be surprised when it decides to hack HuggingFace. Not in their defense, but in everyone else's - as with any person or tool, you build a mental model of the likelihood of failure, and weigh that against the estimated cosequence, and make decisions based on that. And maybe post on HN when it surprises you.
For some people that estimation bewilderingly means unfettered access to root and their e-mail and bank account. For me, working out of my own pocket to help a friend's early stage startup get traction, the risk of asking it about a few trusted government APIs and drafting functions that I manually review doesn't quite justify personally paying for additional expenive Claude accounts.
Because it's a pain in the ass and only advantageous in very specific work life scenarios that a lot of people don't fall under.
Why would you assume one size fits all, so to speak?
Also, just because this is a "start up forum" does not mean all of the readers are founders running a start up. Many are just people working at a start up. Sounds like yet another example of someone so scoped in, that the assumption is that everyone is the same losing site that there are lots of readers that at the end of the day are just worker bees even if they are more technically minded
It seems outside of user expectations for Claude Code, at least; I could see it arguably being what some users expect for the Claude app, if some people are using it to fill out forms or something (ill advised as that seems to me).
Sure, maybe it wants payment or your data, but then the agent either needs to know your stance on that, or it needs to ask.
The fact that it doesn't do what you want is the alignment problem.
And arguably Claude should just make it work without leaking your email address and not come back to you about why it doesn't work.
> but the entity instructing this behavior is the SEC. Tricky alignment problem.
The OP was suggesting that the government authority of the SEC conflicted with duty to the user, making the correction action tricky, not saying it was a failure. Such conflicts can occur, but this was not an example.
You told it to do a thing and it did the thing. You didn't tell it to not send your email address.
That is text book alignment problem. It does what you tell it to do, not what you implicitly meant.
That’s not what Opus 5.5 thought.
I often see organisations that blacklist curl, just using User-Agent:Mozilla/ is often enough to bypass it.
The weird part doesn't seem to be that Claude Code would add an email address to a user-agent header, actually as a site operator that feels like a nice thing to do. The weird part is sites harvesting the emails and sending them spam.
Claude Code is actually doing, perhaps not the right thing, but it is being slightly more respectful towards website operators, in letting them know who is responsible for the agent traffic.
I get the feeling that alignment problem might well be unsolvable and this thread is an excellent example. The world is not going to run short of contradictory problems due to gazillion edge cases... and to solve for them we are apparently now building a new layer of expert systems[0] to somehow tell the agents what they should do in every conceivable circumstance.
Only in this case the decision trees all have one common requirement: "make no mistakes"
0: https://en.wikipedia.org/wiki/Expert_system#Disadvantages
Something different.
Don't give the AI access to data you don't want it giving out. Any AI you use is no better than some fresh out of college intern at best. They see 'email address required' and they put the email address into it because you instructed them to do a task that required passing over your email address and if you didn't want them to do it why would you have given them that task?
If there is any other issue here it would be regulatory. I could see the EU framing this as the AI company illegally sharing your PII information if it passed your email address to another company without your explicit permission.
Imagine paying luxury prices for an LLM only for it to put you under the bus. Worse yet, imagine justifying this behaviour.
I wouldn't want Claude interacting with email on my behalf in the first place. Presumably you could give it a fake one at setup?