News for hackers. Stuff that matters.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @12:00PM (#50534425)
Confirmed. This is 17 TB4 tasks and 45 TB2.1 tasks. Why?
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @12:03PM (#50534473)
We want to see how performance diverges between TB2.1 and TB4 while keeping costs manageable, since running the entire dataset is really expensive.
Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @12:07PM (#50534538)
How were the specific questions chosen?
Re:In 27,280 coding agent runs, 64% of inference spend went ... by Edward40 on Tuesday November 17, 2026 @01:07PM (#50535581)
You can see the task in https://hub.harborframework.com/datasets/frontier-harness-ev...

We kept the category unified so we have tasks for every domain like ML , science and others.

Re:In 27,280 coding agent runs, 64% of inference spend went ... by seizethecheese on Tuesday November 17, 2026 @01:44PM (#50536192)
[delayed]