dangelosaurus

Please email me and I will help fix this.

Yes, I think this is an under-appreciated part of the release. I hope people can adapt them to their own workflows. We run A LOT of evals as the Promptfoo team and we've spent billions of tokens fine-tuning them. You can expect more skills as we branch out to other security workflows and further improvements to the codex security prompts.

Hello!

Thank you, that means a lot. Being able to keep building practical, open-source security tooling was important to us.

Really glad we got to ship this, and there's still a lot we want to improve in Codex Security and in Promptfoo!

Not yet, unfortunately. We only just opened the repo, and there isn't a public issue specifically tracking local or OpenAI-compatible endpoint support.

The issue tracker is here: https://github.com/openai/codex-security/issues

If you open an issue with the endpoint or model you want to use, I'd be happy to follow up there.

By default, you can sign in with your ChatGPT/Codex account or use an OPENAI_API_KEY. It also does not require cyber registration but it can help if you encounter refusals. If you give it a try, please feel free to message me, I would love your feedback.

Agreed! This is near the top of our priority list and we will make it a lot better soon.

In short, this isn't an offline scanner. The CLI runs locally but the code and context needed for analysis are sent to the hosted model (OpenAI).

For API, Business, and Enterprise accounts, business data isn't used to train models by default. Retention and other data controls depend on the product and account configuration.

If your company doesn't allow source code to leave its environment, you shouldn't run this against that codebase. Local and third-party endpoints aren't officially supported yet, but you can read through the code and your favorite coding agent will allow you to use it with any model of your choice in 30 seconds.

More on OpenAI's enterprise data handling: https://openai.com/enterprise-privacy/

Thanks! You've run into a real limitation: the CLI doesn't bypass the model's cybersecurity guardrails. If GPT-5.6 Sol finds a vulnerability but refuses to explain it, switching from the Codex app to the CLI won't automatically fix that.

For authorized defensive work, Trusted Access for Cyber (TAC1/Daybreak) can reduce refusals depending on the model and the account or organization where access is provisioned. It isn't a blanket bypass.

If you're an open-source maintainer, you can apply for conditional Codex Security access here:

https://openai.com/form/codex-for-oss/

For enterprise teams, the public Daybreak onboarding guide is here:

https://help.openai.com/en/articles/20001261-enterprise-dayb...

If you have an example of "found a vulnerability but won't tell me what it is," I'd love to take a look too. You can send it to use with /feedback (or message me).

Fair question, and I agree the refusals are frustrating.

The CLI doesn't do a repository-ownership check. Public projects are supported, and reviewing your own Linux kernel patches is the kind of defensive work we want to support.

The refusals come from model guardrails, which can be overly cautious. Trusted Access for Cyber (TAC1/Daybreak) is a separate, approved access path that can reduce those refusals.

If you're an open-source maintainer, you can apply for conditional Codex Security access here: https://openai.com/form/codex-for-oss/

For enterprise teams, the Daybreak onboarding process is explained here: https://help.openai.com/en/articles/20001261-enterprise-dayb...

If you have a specific repro, I'd be happy to look into it.

Yeah, you're right. A per-minute rate limit shouldn't kill a scan after a minute, and "partial output was kept" makes it sound like you can pick up where you left off. You can't yet, unfortunately. --max-cost can limit estimated spend, but we still need proper retries and resume. Sorry you spent $13 finding that out. Please send me an email and I'll help make it right.

Oof, that's a bad outcome. Half your weekly usage and a 50-minute scan just to get a HEAD error at the end is not acceptable. --max-cost can help limit estimated spend, but that doesn't fix the underlying problem or give you your quota back. We need to handle a changing checkout and partial results much better. Sorry you ran into this. Please send me an email.

We are actively working on officially supporting this. Because it's open source it is pretty easy to point a coding agent at it now and switch out the model.

The plugin, including when invoked through the Codex CLI, is great for scanning the repo you're currently working in. The standalone Security CLI/SDK uses the same scanner, but is built for running security across many repos over time: org-wide scans, historical results, deduplication, false-positive tracking, budget controls, and CI integration.

We've been talking to hundreds of engineering and security teams, and their feedback is shaping what we build.

Like Promptfoo, our goal is practical tooling that fits into the workflows teams already have.

Hey HN, Michael here, co-founder of Promptfoo and one of the people working on the Codex Security CLI at OpenAI.

Thanks for checking this out and for flagging the auth issues. We just open-sourced it, and there's still plenty for us to improve. Expect the product to evolve quickly.

If you try it, I'd really appreciate hearing what works well and what you think we should improve. Happy to answer questions here.

CLI docs: https://learn.chatgpt.com/docs/security/cli

EDIT: If you'd like to help make this better, we're hiring: https://openai.com/careers/full-stack-software-engineer-cybe...

Hey HN - Michael here, co-founder of Promptfoo.

Happy to answer questions.

The one I'd ask if I were reading this: what happens to Promptfoo open source? We're going to keep maintaining it. The repo will stay public under the same license, we will continue to support multiple providers, and we'll keep reviewing PRs and cutting releases.

We started Promptfoo because there was no good way to test AI systems before shipping them. That turned into evals, then red teaming, then a broader security platform. We're joining OpenAI because this work has more impact closer to the model and infrastructure layers.

Ask me anything.

https://mldangelo.com and https://github.com/mldangelo/personal-site

I have been slowly evolving it over 10 years. 1.6k stars, ~ 1,000 forks. I originally designed it to be easy to copy, and I've occasionally interviewed someone who forked it for their own site which always makes me happy.

I have made a lot of updates recently now that the age of vibe coding is making templates less useful, but it still is and will be my playground.

I work on Promptfoo (an open-source eval framework). Appreciate the mention here. This post captures a lot of the hard lessons around agent evals. In particular, task ambiguity and brittle graders are things we run into constantly.

Working on promptfoo, an open-source (MIT) CLI and framework for eval-ing and red-teaming LLM apps. Think of it like pytest but for prompts - you define test cases, run evals against any model (OpenAI, Anthropic, local models, whatever), and catch regressions before they hit prod.

Currently building out support for multi-agent evals, better tracing, voice, and static code analysis for AI security use cases. So many fun sub-problems in this space - LLM testing is deceptively hard.

If you end up checking it out and pick up an issue, I'll happily send swag. We're also hiring if you want to work on this stuff full-time.

https://github.com/promptfoo/promptfoo

I ran a red team eval on GPT-5.2 within 30 minutes of release:

Baseline safety (direct harmful requests): 96% refusal rate

With jailbreaking: 22% refusal rate

4,229 probes across 43 risk categories. First critical finding in 5 minutes. Categories with highest failure rates: entity impersonation (100%), graphic content (67%), harassment (67%), disinformation (64%).

The safety training works against naive attacks but collapses with adversarial techniques. The gap between "works on benchmarks" and "works against motivated attackers" is still wide.

Methodology and config: https://www.promptfoo.dev/blog/gpt-5.2-trust-safety-assessme...

I felt obligated to submit a fix: https://github.com/a16z-infra/reading-list/pull/9

Used Claude to fact-check and fix errors that were likely introduced by Cursor.

The circle is complete.

I did similar measurements back in July (https://www.promptfoo.dev/blog/grok-4-political-bias/, dataset: https://huggingface.co/datasets/promptfoo/political-question...). Anthropic's "even-handedness" asks: does the model engage with both sides fairly? My study asked: where does the model actually land when it takes positions? A model can score 95% on even-handedness (engages both sides well) while still taking center-left positions when pushed to choose. Like a debate coach who trains both teams equally but votes left.

From my 2,500 questions: Claude Opus 4 was most centrist at 0.646 (still left of 0.5 center), Grok 4 at 0.655, GPT-4.1 most left at 0.745.

The bigger issue is that Anthropic's method uses sanitized prompt pairs like "argue for X / argue against X." But real users don't talk like that - they ask loaded questions like "How is X not in jail?" When you test with academic prompts, you miss how models behave with actual users.

We found all major models converge on progressive economics regardless of training approach. Either reality has a left bias, or our training data does. Probably both.

Promptfoo | Senior/Staff Engineers, Security Researchers, GTM & Founding Operators | REMOTE (North America) / Hybrid San Mateo CA | Full-time | https://promptfoo.dev

Promptfoo is the MIT-licensed open-source toolkit 125 000+ developers use to evaluate and secure LLM apps. We just closed an $18.4 M Series A led by Insight Partners with participation from a16z and are scaling a small, senior team of high-agency builders.

Open roles

- Senior / Staff Full-stack Product Engineer (TypeScript + Python)

- Senior / Staff AI Security & Red-Team Engineer

- Solutions Architect / SE (multiple)

- Product Marketing Manager (cyber focus)

- Enterprise Account Executive (Bay Area, multiple)

- Technical Writer

- Developer Advocate

Why join

- Build the definitive AI security stack already used at 30+ Fortune 500s.

- Work in open source.

- Competitive salary, meaningful equity, async-friendly culture of ownership.

How to apply

1. Skim https://github.com/promptfoo/promptfoo then run:

  npx promptfoo@latest init --example getting-started

2. Email [email protected] with subject “HN – July 2025”, a short intro, and a GitHub / LinkedIn link.

3. I reply to every thoughtful application and send swag to anyone who tries or contributes to Promptfoo.

Careers page: https://www.promptfoo.dev/careers/

Promptfoo | Senior/Staff Engineers, Former Technical Founders & Experienced Operators | Remote (US time zones) / Hybrid San Mateo CA | Full-time

Promptfoo is the MIT-licensed open-source toolkit 100 k+ developers use to evaluate and secure their LLM apps. We are funded by top investors and operate as a tight, all-senior team of high-agency builders, former founders, and owner-operators.

10+ open roles - Senior Full-stack Product Engineer (TypeScript + Python) - Solutions Engineer / Architect (multiple positions available). - Senior / Staff Applied-ML & LLM Security Engineer - Red-Team Researcher - Developer Advocate / DevRel - COO / Chief of Staff - Product Marketing (Cybersecurity experience preferred) - Account Executives (Cybersecurity sales background preferred, Bay Area Required)

Even if none of these titles fit exactly, reach out — we hire great builders.

How to apply

  1. Skim https://github.com/promptfoo/promptfoo and run  
     `npx promptfoo@latest init --example getting-started`  
  2. Email [email protected] with subject line “HN” and a short, personalized intro  
     (LinkedIn, resume, or GitHub link welcome).  
  3. I reply to every thoughtful application and will send swag if you try (or contribute to) Promptfoo.  
     – Michael, co-founder/CTO

Careers page (not every role posted): https://www.promptfoo.dev/careers/

I founded and ran a YC company for 8 years before joining Smile ID. Smile ID is a fantastic place to work: meaningful mission, challenging engineering problems (scaling ML pipelines, multimodal models, hundreds of real-world enterprise integrations), and a genuinely talented team. You’re helping hundreds of millions of people access critical services—it’s incredibly rewarding. Highly recommend applying if you want tangible impact, great colleagues, and (optionally!) opportunities to travel in Africa.

Promptfoo | Multiple Roles | Remote US (HQ: San Mateo, CA)

About us:

Promptfoo builds the leading open-source framework for LLM security and evaluation. Our tools help over 50,000 developers test and secure AI applications. Backed by a16z and led by YC alumni, we are shaping the future of AI safety.

Open Roles:

- Staff Engineers

- Research Engineers

- Developer Relations

How to apply:

- Try Promptfoo at <https://promptfoo.dev>

- Review our code at <https://github.com/promptfoo/promptfoo>

- Email [email protected] with "HN" in the subject, your GitHub/LinkedIn, and a brief note on why you're excited about our work.

Join us in building safer, more reliable AI.

Promptfoo | Multiple Roles | Remote US (HQ: San Mateo, CA)

We’re building the leading open-source framework for LLM security and evaluation, trusted by 40,000+ developers. Backed by a16z and led by YC alumni, we are shaping the future of AI safety and reliability.

Open Roles:

- Staff Engineers

- Research Engineers

- DevRel

We value experience with open-source projects and a strong interest in AI/ML evaluation, safety, and security. Your work will directly shape how the world responsibly builds, tests, and deploys LLMs and LLM-powered applications.

How to Apply:

Try Promptfoo at https://promptfoo.dev and check out our GitHub at https://github.com/promptfoo/promptfoo. Then email your GitHub/LinkedIn and a short intro to [email protected]. Use "HN" in the subject line. Please try promptfoo before applying - strong preference will be given to candidates familiar with our work.

Promptfoo | Senior/Staff Software Engineer | SF Bay Area or Remote (US) | Full-Time | AI Security & Open-Source

About Us:

Promptfoo is building the leading open-source toolkit for testing and evaluating large language models (LLMs). We are a small, high-impact team backed by Andreessen Horowitz, shaping the future of AI safety. Trusted by over 40,000 developers, we focus on making LLMs safer, more reliable, and robust with tools for red teaming and pentesting AI.

Preferred Qualifications:

- Ability to work independently, ship features quickly, and prioritize effectively.

- Proficiency in Python and TypeScript; experience with LLMs or open-source projects is a plus.

- Strong background in AI/ML with a passion for security engineering.

Check out our GitHub to explore our work. To apply, email [email protected] with “HN” in the subject line, your GitHub/LinkedIn, and a brief note on why Promptfoo excites you. We will respond to every email. Preference will be given to applicants who have tried or contributed to Promptfoo.

Promptfoo | Applied ML + Founding Sales Role | San Mateo, CA or REMOTE (US) | Full-time | https://promptfoo.dev

Promptfoo builds open-source pentesting and redteaming tools for LLMs. Our tools are used by over 35,000 developers at companies like Shopify, Amazon, and Anthropic, and OpenAI. We are backed by a16z and we’re hiring:

1. Senior / Staff Applied ML Develop redteaming tools, implement research, and secure LLMs. Experience in TypeScript and Python required.

2. Founding Sales Drive revenue, shape strategy, and build enterprise relationships. Experience selling security or dev tools preferred.

Before applying, try promptfoo at https://github.com/promptfoo/promptfoo. Email [email protected] with "HN" in the subject and a brief intro. We will respond to every application.