Well I'm not too familiar with Rust so I can't gauge correctness, but I do have the Llama 2 13B NewHope fine tune loaded (which is afaik tuned for python coding), so I gave it and 3.5-turbo the same random post request question.

3.5's result: https://chat.openai.com/share/9e1aafd3-631c-4c13-80f6-f99c88...

NewHope's result: https://i.imgur.com/dfACQC3.png

If you have any ideas for a more comprehensive test let me know and I'll try to run it. Giving it some existing code to fix up or change is usually more of a typical use case for me anyway.

Interestingly, they have withdrawn their model because they discovered that test data leaked into the training data. Quantised versions are still available on huggingface from others though.

https://github.com/SLAM-group/newhope

Yeah it seemed suspiciously high for HumanEval and it only ranks 14th for JS and 7th for Python on other benchmarks now: https://huggingface.co/spaces/mike-ravkine/can-ai-code-resul...

WizardCoder is a bit of a problem since it's not llama 1/2 based but is its own 15B model and as such the support for it in anything practical is near nonexistent. WizardLM v1.2 looks like it may be worth testing out.

All of the LLama 2 fine tunes I've tried out so far have weird issues though. Saying unrelated things at times, ignoring parts of the conversation and such. Could be fine tuning or prompt template goofs or Llama 1 may actually be a more self consistent base model overall.