joshonyc

Very much disagree, to me it clearly shows the gap between Open Source and the frontier. You can not do real code work with open source models right now.

Especially since this benchmark is still mostly DeepSWE, and the only reason the open source models are doing decent is because much of the information hasn't changed. The open source models have been benchmaxxed to death.