News for hackers. Stuff that matters.

A comment on German AI consortium releases Soofi S, an open 30B model that tops benchmarks.

Re:German AI consortium releases Soofi S, an open 30B model ... by jbellis on Thursday July 16, 2026 @03:26PM (#48939105)
It tops benchmarks because it uses them in its training data. https://x.com/eliebakouch/status/2077425801633427919
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by sva_ on Thursday July 16, 2026 @03:31PM (#48939160)
https://nitter.net/eliebakouch/status/2077425801633427919

That is very disappointing.

[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by Tostino on Thursday July 16, 2026 @04:24PM (#48939752)
That's just embarrassing. It keeps happening though.
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by sajithdilshan on Thursday July 16, 2026 @03:46PM (#48939344)
Disappointing. Sounds like Volkswagen 2.0
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by krackers on Thursday July 16, 2026 @04:20PM (#48939689)
training on the test set is all you need
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by ch4s3 on Thursday July 16, 2026 @05:27PM (#48940486)
It's ready for law school!
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by AtlasBarfed on Thursday July 16, 2026 @05:41PM (#48940652)
scrape the scrapers... all's fair...
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by fxwin on Friday July 17, 2026 @06:36AM (#48945614)
looks like they have since fixed the dataset, hope they release an updated report (either without the leaked evals or a new checkpoint/weights)

https://huggingface.co/datasets/AIML-TUDA/QA-base/commit/2f9...

[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by istvan0 on Friday July 17, 2026 @11:27AM (#48948555)
Reminds me of: https://en.wikipedia.org/wiki/Volkswagen_emissions_scandal
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by amai on Friday July 17, 2026 @11:32AM (#48948614)
It seems they corrected this already: https://nitter.net/jphme/status/2077468512025477226#m
[ Parent | on HN ]
Re:German AI consortium releases Soofi S, an open 30B model ... by jbellis on Friday July 17, 2026 @01:27PM (#48949937)
I guess you didn't read ellie's reply directly underneath that?
[ Parent | on HN ]