News for hackers. Stuff that matters.

A comment on Mistral 7B, the most powerful language model for its size to date, Apache 2.0. Parent comment

Re:Mistral 7B, the most powerful language model for its size... by mufasachan on Thursday September 28, 2023 @12:56AM (#37685391)
One explaination https://x.com/yampeleg/status/1707127722743325106?s=46&t=Cxa...

I would be curious what does he mean by "semi-automated system for detecting benchmark leaks. " though.

[ Parent | on HN ]
Re:Mistral 7B, the most powerful language model for its size... by brucethemoose2 on Thursday September 28, 2023 @10:25AM (#37690051)
AFAIK, such tests just feed the model chopped up bits of the evaluation data as raw strings with zero temperature. If it completes them verbatim, its probably in the training dataset.
[ Parent | on HN ]