Trapais

Have you tried programming in something other than notepad.exe?

Modern IDEs have auto complete so impressive, that you don't need to type much anyway. I type several symbols then stop to let the editor suggest the rest of the word. Can I touch type it myself? Yes. Do I want to? No. There wouldn't be much difference if I typed slower.

NB. When I say "modern IDEs" I mean anything since around Visual Assist, which sorted suggestions not alphabetically, but depending on context. I don't even mean copilot.

For comparison, here's 8B [Nemotron](https://huggingface.co/nvidia):

> 1,024 A100s were used for 19 days to train the model.

> NVIDIA models are trained on a diverse set of public and proprietary datasets. This model was trained on a dataset containing 3.8 Trillion tokens of text.

People say lots of stupid shit. If there is no code or even paper, there is no reason to believe. Beating benchmarks requires something more than a blind faith.

>Making useful models is the goal.

Sure, training datasets for pythia is useful. The Pile was used in lots of models. However it's hardly relevant that pythia itself was trained on pile. They live separate lives.

Having just weights already allows making results that are incredibly useful(you don't need original dataset for flash attention, or tuning foundation model into the chat model).

Point is: Having both doesn't make released model more useful.

>Do you actually have any experience doing this? Have you ever fine tuned models or tried to change architecture or put a piece of one model into another?

Yes on both finetune and "changing" architecture: with adapters and similar approaches you don't need to retrain everything from scratch after modifying the guts of the original architecture up to your liking, you just need to not stir it up too much. Training on the task at hand is sufficient.

No, I haven't glued parts of existing models together(ensemble doesn't count)

>Are there any true open-source LLM models, where all the training data is publicly-available (with a compatible license)

Mamba has a version, trained on publicly available SlimPajama. RedPajama-INCITE was trained on non-slimmed version of the dataset(it's only one dataset).

I'm not sure if training scripts are available.

Pythia definitely has scripts. However it was trained on the pile, so you have to find books3 on your own.

Also I believe LLM360 is an explicit attempt to do it with llama.

>Is training nondeterministic?

Correct. Torch documentation has a section on reproducibility of a training.

OK. Where is your reproduction of Pythia trained from scratch? Or MPT? Or Amber? Shall we play a game where you give paper regarding pretraining (and we are not taling about puny models based on wikitext2) I give you a paper based around finetuning and we'll see who run out of papers first?

You can grep for bad words. What you can't do(unless hoops are jumped through) is to verify that weights came from the same dataset. You can set the same random seed and still get different results. Calculations are not that deterministic. (https://pytorch.org/docs/stable/notes/randomness.html#reprod...).

>I am overall skeptical that this is true in the case of LLMs

This skepticism seems reasonable. EleutherAI have documentation to reproduce training (https://github.com/EleutherAI/pythia#reproducing-training). So far I haven't seen it leading to anything. Lots of arxiv papers I've seen complain about time and budget constraint even regarding finetunes, forget pretraining.

Propaganda. DoD already uses hollywood for propaganda: say nice things about Uncle Sam, let America save the day once again, and Uncle Sam will let you play with his toys. Now they have access to tool that can write very smart comments. If you think propagandists will not use SoTA LLM for propaganda, either you are fool or they are(They probably did it already anyway).

General writing. Want to implement a new rule? Ask GPT to reword it so even idiots can understand. Ask if it contradicts existing rules(they should have similar systems already, but GPT is smart).

Combination of above. "I want to develop a gas chamber. How do I announce it to the public so it looks like I am doing humanity a favor?"

Possible reaction. "I developed a gas chamber and announced it as 'Overpopulation relocation centers'. How will democrats/republicans will react to it?"

Targeted writing. "I developed a gas chamber and announced it as 'Overpopulation relocation centers'. It seems democrats/republicans don't like it. How do I reword it so they do like it?"

Deep fakes, image. "Terrorist Terro Rist wants Inno Cent to be dead. Create an image of Inno Cent with a bullet hole in his forehead so Terro will go celebrate a victory and we'll shoot him for real, here's a photo"

Faking text. "Here's writings of Terro Rist. Write 'meet me at 7:00 at central square in his style, keep using his punctuation, do the same grammar errors, etc"

Useless Voodoo GPT is not Good For but Still Will Be Used Because They Have Access to It. "Here is a photo of Inno Cent who was shot dead. What gun could have been used?"

Spying. "We want to see all requests coming from China or Russia paired with IP", "All requests with words 'President' and 'Murder' should be forwarded to this email at once", etc

They will find several uses.

Looks like longformer to me. They just renamed "global attention" into "attention sink" and removed silly parts(distilled attention) and BERT parts([CLS] saw all N tokens, there is no need for BOS to see all tokens)

> In my opinion, formed from over two decades of Linux, a piece of hardware having a libre driver written for it is the exact indicator of what can be relied upon to "just work".

Then this opinion can be discarded as not grounded in reality. In order to run pytorch at high speed I need to do... basically nothing. It just works.

Many things can be said about ROCm support for 7900XT, none of them are positive. "It just works" drivers from AMD come several months too late.

I have doubts it was extensively trained on German data. Who knows about GPT4, but GPT3 is ~92% of English and ~1.5% of German, which means it saw more "die, motherfucker, die" than "die Mutter".

(https://github.com/openai/gpt-3/blob/master/dataset_statisti...)

That's a very fancy way with lots of fancy words to say "I have no idea how NN work, but if I sound smart maybe ppl will not figure out how stupid I sound". Well, you sound stupid and nothing what you said makes sense.

Delineate? Diffusion models can't be further away for delineating. They literally work by throwing random shit at the wall.

Input set? There is no input set once training is complete.

Human creativity? In stock photo industry? What next? "How I write while loop instead of for loop and I have achieved the nirvana?"

Maybe they should support their modern consumer cards in ROCm. Maybe their ROCm documentation should not suck balls.

I'd say there is a reason AMD is a laughing stock in ML, but it's incorrect. There's not a reason, there are tons of reasons to never touch it.

I will believe that it's open source the moment weights are downloaded on my computer and I don't need to summon DAN for using them: their goal is to make AI "safer", which is corporate for "heavily censored, but just in the way we like it".

If you say "BLOOM" and "good" in the same sentence, you have absolutely no idea what you are talking about. BLOOM is a trash, poster boy for why number of parameters doesn't matter.