News for hackers. Stuff that matters.
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by visarga on Tuesday October 31, 2023 @01:51AM (#38080778)
Great work, may I suggest more analysis features?

- example summary, for better topic embedding

- RAG based summary, to have the model critically assess its training data distribution and answer questions on it; to bring together information sitting in separate examples

- named entities, for knowledge base; maybe it helps with fact checking later

- implicit tasks present in the text, what are the tasks a LLM could learn from a given example?

- chain-of-thought augmentation, to bring out implicit deductions and reduce information fragmentation; it has been shown in the Phi-1.5 paper and Orca that synthetic CoT datasets are superior source materials

What data fragmentation? Look at the Reversal Curse paper. Models that train on "A is the father of B" fail to generate "B is the son of A". This kind of connection needs to be explicitly added, and would improve task solving as well.

Training on purely organic data is not good enough anymore. All powerful models train on a mix of organic and synthetic data, some models on 50-50 proportions, like the web+synth variant from Phi-1.5.

The main idea is to go deeper into the raw data, to infuse it with insight. LLM dataset preprocessing is going to be expensive, comparable to training costs, but the results are worth the effort.

Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by sorokod on Tuesday October 31, 2023 @02:39AM (#38081032)
"B is the son of A" doesn't follow from "A is the father of B".
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by TeMPOraL on Tuesday October 31, 2023 @10:44AM (#38085627)
I feel this is usually a hugely asymmetric problem. The other example I've seen is the model being able to easily complete "The color of the sky is" with "blue", but then failing to complete "Blue is the color of" with "the sky". I say, d'uh, why would it?

If you take the training data, or in general, imagine taking all that humans ever wrote or spoke in English to date, you'll expect to find an overwhelming amount of cases where "The color of the sky is" ends with "blue". However, "Blue is the color of" can easily have a hundred thousand different plausible completions, and "the sky" won't even be one of the more likely ones. In the absence of additional context that strongly hints at the answer, one should NOT expect a properly working LLM to frequently propose "the sky" as completion to "Blue is the color of".

Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by visarga on Wednesday November 01, 2023 @12:33PM (#38100769)
you're right, replacing son with child
Re:RedPajama v2 Open Dataset with 30T Tokens for Training LLMs by zhangce on Tuesday October 31, 2023 @02:12AM (#38080911)
Thanks for the suggestion! We will add this in the pool of features for future release. (We are currently running the current 40+ annotations on the `tail` partitions).

If you are interested in contributing the code for these features, feel free to do a PR to https://github.com/togethercomputer/RedPajama-Data! Otherwise we will try our best effort implementation :) but we hope that this can become a community effort

(feel free to created more issues on github for us to keep track. I created one for this https://github.com/togethercomputer/RedPajama-Data/issues/76)