madiator

Check out OpenThoughts. It has a widely used dataset, a model that beats the deepseek's smaller reasoning models, and a paper that talks in detail about the data curation methodology.

https://www.open-thoughts.ai/

I wrote about this recently as well: https://madiator.substack.com/p/finetuning-is-so-back

Congrats! Great work, Dan! Indeed, Terminal bench is awesome :)

Right. The title conjures dreamy images in my mind that I was dying to see images!

That's the hope, but that's never how things go. Of course researchers and research will be tremendously affected, and that's a shame.

Yeah this was quite an interesting finding, given how Mistral has marketed this (OCR solved!)

The blogpost is linked here: https://news.ycombinator.com/item?id=42826392

Thanks! We created bespoke-stratos-32B - let me know if you have any questions.

Synthetic data got a bad reputation last year, but it is now an important component for all modern LLMs! In fact, we had also trained one model for -- ironically -- detecting hallucinations, and it was also trained on synthetic data.

Say if you have some PDFs, and want to generate questions and answers to test your RAG pipeline, that's synthetic data! Distillation is mostly synthetic data and works great as well!

Our hope is that this becomes steroids rather than LSD for LLMs :)

We are working on it!

Not sure why you are so upset about a small and neat study ("please, please stop"). If you ask it to summarize (without feeding the entire bible), it needs to know the bible. Knowledge and reasoning are not entirely disconnected.

For the specific form of hallucination, which is called grounded factuality, we have trained a pretty good model that can detect if a claim is supported by a context. This is super useful for RAG. More info at https://bespokelabs.ai/bespoke-minicheck.

Now, it will be nice to get the code snippets for these!

There are several types of hallucinations, and the most important one for RAG is grounded factuality.

We built a model to detect this, and it does pretty well! Given a context and a claim, it tells how well the context supports the claim. You can check out a demo at https://playground.bespokelabs.ai

That's great, but it did not really write the program that the human asked it to do. :)

I am sure most people never asked these questions to a human doing this research.

I will be typing this comment on my Nokia's keyboard.

Yeah it wouldn't fit. GPT3 is 175B params, so even if you use 8 bit for each weight, you need 175×10^9÷2^30 = 163GiB of memory.

I am the author of the blog and I want to thank for all the feedback here. Because I now do realize the article starts with a high premise but fails short. Will do a better job next time!

What I meant to talk about is as follows: we have engineered away all sorts of discomfort from our lives and I think that's bad. So (1) be aware of this, (2) seek some discomfort, and (3) if you run into discomfort, take it in a positive way (this I didn't convey in the article).

But yeah I don't mean to say chop of your limbs! Not sure how people are reaching that conclusion.

Thanks HN!

Here's an example I can think of. Suppose you have a bunch of text documents, and you know that some documents are similar but not identical (e.g. plagiarized and slightly modified). You want to find out which documents are similar.

You can first run the contents through some sort of embedding model (e.g. the recent OpenAI embedding model [1]), and then apply LSH on those embeddings. The documents that have the same LSH value would have had very similar embeddings, and thus very similar content.

[1] https://beta.openai.com/docs/guides/embeddings