It seems here the guard rails at failure were the llm users right? Whatever guard rails you can think may be useless against the superior human stupidity.
Also, what's the LLM use policy at the SD-6?
Prompts are just weights on a graph traversal. They don't guarantee anything. The LLM does not "understand" the prompts and so it cannot fully adhere to them. They only improve the liklihood it will output what you want.
Never ever ever give an LLM access to something you can't afford to break. And stop thinking of them like people.
I agree the guy is an idiot for trusting these AI models.
OTOH AI companies keep running and marketing their services with zero accountability for mistakes.
People have been drinking too much hopium they have lost touch with reality.
Everyone needs to properly understand these tools before they use them for anything serious.
Also, it's not a "confession". It's an LLM stringing together some tokens that form words trying to make a pleasing-sounding answer. Plus, the first sentence and the context implies that someone gave it a prompt that told it to never guess around but get stuff done. OP branding this as a confession tells you everything you need to know: total and absolute failure of guard rails, but these guard rails can not be expected to be in an LLM.