I built a vulnerable app and spent $1,500 seeing if LLMs could hack it

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by ben_w on Thursday June 04, 2026 @04:45AM (#48395937)

While this is a perfectly reasonable thing to expect when the models are competent enough, half the conversation on places like Hacker News are about all the times an LLM has produced garbage that was harmful to a business either by hallucinations, by deleting something critical during the work, or by hitting some endpoint way too often and denial-of-servicing it.

Right now, the software guardrails in LLMs are useful for the same kinds of reasons factories have hardware guardrails: to reduce the rate at which errors become "incidents".

Just because they sometimes delete the production database rather than sometimes spilling a thousand tons of incandescent molten metal over a factory floor, doesn't mean LLMs are safe enough to be used the way they're actually being used.

https://simonwillison.net/2025/Dec/10/normalization-of-devia...

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by throwway120385 on Thursday June 04, 2026 @10:27AM (#48399176)

I think you're assuming too much care. Right now they haven't adopted that business model because they don't see it as a viable business model. As soon as they realize that they can lock certain categories of query behind a different subscription they will do that. We saw the same thing with streaming services and basically every other kind of online service -- small, singular subscription followed by a gold rush and then suddenly there's an upcharge for access to every other publisher's catalog of movies.

[ Parent | on HN ]

Re:I built a vulnerable app and spent $1,500 seeing if LLMs ... by ben_w on Thursday June 04, 2026 @03:39PM (#48403623)

That kind of thing is basically why I wrote the opening clause of the first sentence.

i.e., yeah, probably.

[ Parent | on HN ]