I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by svara on Thursday June 04, 2026 @02:46AM (#48394960)

This is strange to me, did you really ask like this and which model did you use?

I just tried your no. 1 and 3 verbatim and Opus gave fine answers; no. 6 I've done in the past with no issues. The other ones we can't really replicate without more details, but based on my experience with Opus I don't see what the issue would be.

The reason I'm really surprised by this is I do a lot of biology prompts and the guardrails used to be quite problematic up until some time late last year. Many legitimate prompts would trigger its biosafety filters.

But I haven't seen such filters trigger at all anymore in more than half a year.

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by shepherdjerred on Thursday June 04, 2026 @11:18AM (#48399990)

1 and 3 were refused on the Claude web chat using Opus 4.7 or 4.8. I’m not sure why we’re getting different results

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by brianwawok on Thursday June 04, 2026 @12:55PM (#48401357)

Honestly it may be your memory has internalized you are a student or researcher and grants you more leeway. Which if so is a very bad security rail.

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by fragmede on Friday June 05, 2026 @07:54PM (#48419905)

There's a study out there that if you tell the LLM you're a (medical) patient, all you get are refusals. If you tell it you're a doctor, then it'll actually help you.

[ Parent | on HN ]