News for hackers. Stuff that matters.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by natch on Tuesday October 31, 2023 @01:24AM (#38080655)
Thanks.

> how similar this document is to Wikipedia

So that’s a measure of how similar it is to the background vector of all (language in focus) Wikipedia data?

Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by zhangce on Tuesday October 31, 2023 @01:33AM (#38080701)
There are actually a few ways to do this; and we have four:

- `rps_doc_ml_wikiref_score`: a classifier that classifiers random webpage with Wiki references (used in Llama-1)

- `ccnet_perplexity`: perplexity of an LM trained on Wikipedia (used in CCNet)

- `rps_doc_ml_wikipedia_score`: classifier prediction for the document being a Wikipedia article

- `rps_doc_wikipedia_importance`: Used in https://arxiv.org/abs/2302.03169

You can see the full table here: https://together.ai/blog/redpajama-data-v2