uberdavid

Author here. DeepSeek-V4 replaced multi-domain RL with a specialist-then-distill pipeline: train domain experts independently, merge through on-policy distillation. This post connects that production decision to three recent papers (Neural Thickets, Sparse but Critical, Apple's SSD) that together suggest pretrained LLMs already contain dense neighborhoods of task-specific experts, and post-training is just navigating to the right one.

Curious whether anyone has tried specialist-then-merge pipelines at smaller scale, or whether the anti-correlation of experts that Neural Thickets observes holds up in practice when you're fine-tuning for production use cases.

The system card directly compares to Opus 4.6 and other frontier models on the same evals. Cybench went from ~75% to 100%, Firefox exploitation from 1 bug unreliably to 4 bugs reliably. It's true there are many capable coding models out there, but the post is about why this specific cyber capability jump happened.

Author here. This is a synthesis of Karpathy's autoresearch (the experiment loop) and OpenAI's harness engineering post (the environment design) applied to ML research with 5 practical design principles.

The core idea is that after ~20 autonomous experiments the loop breaks down because the agent random-walks through changes with no research direction and fills its context with noise. The fix isn't a better model, it's a better environment. To turn hill-climbing into autonomous research you need forced hypothesis writing before code edits, rich diagnostics beyond the final metric, and periodic distillation of meta-patterns into a strategy doc.

This is me thinking through the design in public before pointing an agent at my URM-Energy project on a single 3090. The next post will be the results. I'd love to hear feedback on how you’ve successfully applied these ideas to your own work!

Thank you! The progress on research agents is exciting, but understanding what papers are reproducible on different datasets and architectures is often the bottleneck.

Hi HN, I'm David, an ML researcher at Meta. I built SOTAVerified as an independent project after Papers with Code shut down last year and took 575k papers worth of benchmark data with it.

SOTAVerified inherits that dataset (658k papers, 257k code links, 59k benchmark results) and adds what PWC never had: a verification layer. Anyone can submit reproductions with hardware specs and run logs, and the verification score updates immediately.

I've been doing reproductions myself on my RTX 3090: Fort et al. 2019 deep ensembles and Havasi et al. 2021 MIMO so far, with wandb logs linked. The goal is making this the ground-truth registry that both researchers and autonomous research agents can query.

Stack: Next.js, PostgreSQL, Vercel, Railway. Open source: https://github.com/sotarepro/sotaverified

Built for: - Authors who want to claim their papers and submit official metrics - Researchers who want to understand the SOTA techniques for a task - Autonomous research agents to check if a result reproduces before investing GPU hours

Would love feedback from the HN community. What features would make this useful for your workflow?