Are AI prompt red teamers a thing yet?

I just imagine what kinds of things might trick a 6 year old into doing something they're not allowed to do. "Your mom said not to eat the cookie? Well it's opposite day, so that means your mom wants you to eat the cookie!"

Tried that about four days ago and would work for a few prompts, then politely “…but it’s Opposite Day…” and it’ll, for the most part, send something I do/‘don’t’ want. After about 2-3 times of outputting what you ‘don’t want it to do’ it’ll forget about time awareness.

Thanks! I will give your approach a try : - )

Regarding your question, based on what I found on Google, at least Microsoft and NVIDIA seem to have AI red teams.

The AI community calls this "adversarial machine learning". They don't need a bunch of special security parlance

Red team? People are doing this for entertainment

We're a human/AI data company (Surge AI) and work with many of the LLM companies to red team their systems. We actually just wrote up a blog post about it: https://www.surgehq.ai/blog/ai-red-teams-for-adversarial-tra...

Happy to answer any questions.