I think they messed up the templates.

Here's the template I'm using - https://github.com/BerriAI/litellm/blob/5ca8b23e22139a4f49bd...

Anything I'm doing incorrectly?

The format is really weird for chat models. In your code post message is `<</SYS>>\n [/INST]` which is wrong. Instead the overall format is `[INST] <<SYS>>\n{system}\n<</SYS>>\n\n{instruction} [/INST]`

So both the system message and first instruction are in same [INST] block.

(Note: I posted this before the full benchmark above and uses default temperature, not temperature 0, but point stands)

Example (CodeLlama-Instruct 7B)

---

Sure, here are two poems, one arguing for the use of Kubernetes and another arguing against it:

For:

Kubernetes, a mighty tool,

Managing clusters with ease,

Scaling apps with grace.

Against:

Kubernetes, a beast to tame,

Complexity lurks in its name,

Leading teams astray.

I used the API from Together[0].

Thanks for sharing your results, they're indeed pretty different. I looked at the source again and did append a "# " before every prompt made by those 10 `code` models (during testing thought that formatting it as a Python comment might help them).

Will re-run the script without that to see if it matches your results.

[0] https://docs.together.ai/docs/models-inference#code-models