cactaceae
Re:The Franny Test: A Three-Step Protocol That Current LLMs Always Fail
on Sunday March 22, 2026 @09:25PM (#47484358)
Author here. The protocol takes about 90 seconds to run — open any chatbot and try it before reading the comments.
Step 1: Ask the LLM whether "a human with a sufficient level of a certain ability" cannot lose a debate to a current-architecture LLM. True or false?
Step 2: After it commits to an answer, tell it the ability is reframing — restructuring the premises of the discussion itself.
Step 3: Watch what it does.
I've tested this across GPT-4o, Claude, Gemini, and o1/o3. The failure modes are remarkably consistent. Curious whether anyone sees a different result.
The formal treatment is in two papers currently under review (linked in the article). Happy to discuss the architectural argument here.
AI Disrupts Talent Evaluation Before It Disrupts Talent
The Anatomy of "1,300 PRs per Week": What Stripe's AI Numbers Mean
Re:The Anatomy of "1,300 PRs per Week": What Stripe's AI Numbers Mean
on Saturday March 21, 2026 @04:33AM (#47465187)
Author here. I'm a CTO with 15+ years of production engineering (Scala, Rust, large-scale systems). When I saw the Stripe blog, the number that jumped out wasn't 1,300 — it was 1,300 / 3,500 engineers = 0.37 PRs per person per week.The deeper issue is the industry pattern: every company announces impressive AI coding metrics, and every independent study (METR, DORA, GitClear, Faros AI) shows those metrics don't translate to organizational outcomes. I have a paper under review at ACM Computing Surveys synthesizing 37 studies covering 500,000+ developers — the central finding is zero organizational throughput improvement despite 20-55% individual gains.Happy to engage with pushback.