News for hackers. Stuff that matters.
Re:Training great LLMs from ground zero by ganeshkrishnan on Wednesday March 06, 2024 @07:18PM (#39623541)
OP mentions the failure rate of GPUs as "If this were in GPU land, it would have failed within the first few days for sure.".

In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4.

GPUs have memory constraint issues but if you can plan and work around it, I havent seen it crash in real life.

Re:Training great LLMs from ground zero by teaearlgraycold on Wednesday March 06, 2024 @09:12PM (#39624336)
Ha! We’re also committing great sins of computation against T4s at our company. Hopefully, as I learn, things get less janky.
Re:Training great LLMs from ground zero by gwern on Wednesday March 06, 2024 @09:13PM (#39624346)
> And we are using the older Tesla T4.

That's an undemanding and well-debugged chip by this point (6 years ago!). So you aren't experiencing any of the pain people using A100s or H100s (never mind people who have to stand up clusters with B100s soon) are going through now.

Re:Training great LLMs from ground zero by icpmacdo on Monday March 11, 2024 @01:21AM (#39665136)
> never mind people who have to stand up clusters with B100s soon

Gwern(or anyone else) do you have any resources on this?

Re:Training great LLMs from ground zero by shrubble on Thursday March 07, 2024 @12:59AM (#39625501)
Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...
Re:Training great LLMs from ground zero by flybarrel on Thursday March 07, 2024 @10:52AM (#39630533)
What would be an ideal (or more appropriate) speed?
Re:Training great LLMs from ground zero by shrubble on Thursday March 07, 2024 @04:44PM (#39634920)
Well it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.
Re:Training great LLMs from ground zero by mc10 on Thursday March 07, 2024 @09:43PM (#39637254)
simdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson
Re:Training great LLMs from ground zero by nl on Thursday March 07, 2024 @08:08AM (#39628601)
> 20GB json file… takes 6 hours just to load

Err you definitely should be doing something about that.

20GB on T4s (how many?) isn’t really comparable to terabytes on thousands of A100s.