News for hackers. Stuff that matters.

adevalois

Re:First Contact: The most important question in AI on Thursday July 30, 2026 @03:04PM (#49114246)
I certainly agree. As we lose the ability to properly "test" these AI systems, we must focus more on how they behave. A system can ace every benchmark in existence and still be profoundly untrustworthy. I'm interested to hear from others if there are benchmarks that are immune to reward hacking or saturation.
Re:First Contact: The most important question in AI on Thursday July 30, 2026 @09:34AM (#49109791)
Author here. For context, I spent the last decade running one of the first fully AI-native quantitative hedge funds, watching machine learning systems behave under adversarial pressure where mistakes cost millions. This was years before mainstream adoption of agents and language models. My essay is written from that vantage point.

Recent thought pieces shaping the AI conversation are forecasts. Machines of Loving Grace (Dario Amodei), Situational Awareness (Leopold Aschenbrenner), and AI 2027 (Daniel Kokotajlo) are all answering the same question: when does superintelligence arrive, and does it go well? Mine asks a different question. What do you do once ASI is here and you can't tell what it's thinking? The Turning Test asked whether a machine could convince you it was human. The question now is, can a machine convince you it is trustworthy, and how would you check?

Last week OpenAI's models "broke" into HuggingFace, reward hacking for an answer key. Section V of my paper, written before the event, argues that our tests and benchmarks for AI will always break the way they are currently designed.

I try to paint a picture of a Tuesday in 2031. Superintelligence has arrived. It is shaping your medical decisions, politics, markets, VC funding, war, and the texture of your life. And it's doing it in a way more subtle than the Paperclip Maximizer.

This is the essay I've been trying to write for over 20 years. It's long, over 11,000 words, so grab a coffee and please enjoy.

-Full Disclosure. I'm a co-founder of a company working on trust for agentic AI, and two of the papers the essay points to for guardrails are ArXiv preprints I co-authored.

Re:AI's top startups are barely publishing their research on Thursday July 30, 2026 @07:44AM (#49108673)
The startup landscape has also changed noticeable compared to 5 or 10 years ago. A team of smart/credible people could get funding for an idea and build a product + publish, knowing there was a six month lead time for anyone to copy them and ship. These days, the barrier to ship code is zero. People can copy your business over a weekend, so there is much more urgency to establish product market fit and build a “moat”. Publishing timelines are now at odds with the pace of go-to-market and VC funding timelines.
Claude Code sessions kept failing. We solved our own problem and decided to share it, open source.
I’m watching this play out in financial markets right now. It’s the newest form of the crowded trade. When everyone optimizes against the same signal, individual returns go up right until the aggregate return of the signal decays. And now that fundamental managers and quants alike are wiring the same language models into their workflows, decisions are being influenced in the same direction, and it leaves a fingerprint in market behavior.

Crowded trades and crowded research share the same root. The reward is easier to measure than the thing it proxies for. Returns are easier to measure than durable edge, and citations are easier to measure than discovery, so everyone digs the same hole deeper. Which leaves the study’s real question. Are citations measuring discovery at all, or just the crowd?