I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by Bratmon on Thursday June 04, 2026 @04:24PM (#48404129)

> It’s not because of capability, it’s because Anthropic’s guardrails prevented it from solving the problem.

I'm not familiar with this case, but in general people should be very suspicious about this claim- it is extremely common for an LLM to claim they're not allowed to do something when in fact they're incapable of it.

After all "My code of conduct forbids me from..." is a completion just like any other, and if the LLM can't perform a task, it's usually the best completion.

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by gck1 on Thursday June 04, 2026 @05:43PM (#48405022)

No. Anthropic runs prompts through a classifier that then proceeds to do prompt injection on anything dual-use, which then results in an escalating flag on your account, which increases the strictness of the classifier and volume of prompt injections progressively.

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by SOLAR_FIELDS on Thursday June 04, 2026 @09:33PM (#48406933)

My anecdata from my example demonstrates it’s not the case. I hit the security guardrail, then start a new prompt, asking it to do literally the exact same thing in a different way and without the lead up context, and it happily does it

[ Parent | on HN ]