jasondavis

We curated a library of 60 ways to improve an AI agent, drawn from 279 sources. We keep the results that cut against a technique: 166 of the 662 we cite are mixed or contradicting.

Entries are cataloged across impact type (cost, quality, speed), agent component (model, memory, prompts, tools, etc), how to test it, and sources that we found to explain / provide data.

The library can also be ingested in bulk into your favorite AI coding tool.

Would love to know what's missing or if anything disagrees with what you've seen working. Happy to answer any questions.

We built six agent benchmarks (recommendations, tool routing, text-to-SQL, rag, extraction, multi-turn error recovery), held each one fixed, and swapped only the model: Haiku 4.5 → 5.5 and GPT-5.4 nano → GPT-6 Luna.

What surprised us: the lab's own successors lowered the score in 3 of 12 upgrades, and the other lab's new model matched or beat it in 10 of 12. Prompts and tool descriptions also benefited from significant tuning with these upgrades.

Would love to know if folks have similar experiences and happy to answer any questions.

Thanks sheraz - we certainly don't expect Jenkins to serve as core infrastructure for many of these problems long term.

Absolutely - we (i.e. engineering @ Simon) view Jenkins as crontab that's easier to maintain, to track historical runs, and to keep tabs on failures (with full logging).

Thanks Dan. I actually wrote a piece a couple years back about this as well:

http://drjasondavis.com/blog/2013/09/12/eight-ways-youve-mis...

Can't tell you how many bad a/b tests we ran at Etsy until we figured things out ;)

With today's trends of massive series A rounds, how should entrepreneurs think about staying lean with so much pressure to go big and a prevailing winner-take-all mentality within the VC world? Moreover, what's your take on the series A mega-round?

So true. People talk about failing fast, but recognizing failure is perhaps even a bigger challenge.