Re:Training great LLMs from ground zero
by teaearlgraycold on Wednesday March 06, 2024 @09:12PM
(#39624336)
Ha! We’re also committing great sins of computation against T4s at our company. Hopefully, as I learn, things get less janky.
> And we are using the older Tesla T4.
That's an undemanding and well-debugged chip by this point (6 years ago!). So you aren't experiencing any of the pain people using A100s or H100s (never mind people who have to stand up clusters with B100s soon) are going through now.
> never mind people who have to stand up clusters with B100s soon
Gwern(or anyone else) do you have any resources on this?
Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...
Re:Training great LLMs from ground zero
by flybarrel on Thursday March 07, 2024 @10:52AM
(#39630533)
What would be an ideal (or more appropriate) speed?
Well it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.
simdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson
In my humble opinion, we never had failures of GPU even for large scale training. Our current training batch job is a 20GB json file which takes 6 hours just to load and has been running for more than 15 days with not a hiccup. And we are using the older Tesla T4.
GPUs have memory constraint issues but if you can plan and work around it, I havent seen it crash in real life.