Re:Training great LLMs from ground zero
by shrubble on Thursday March 07, 2024 @12:59AM
(#39625501)
Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...
[ Parent
| on HN ]
Re:Training great LLMs from ground zero
by flybarrel on Thursday March 07, 2024 @10:52AM
(#39630533)
What would be an ideal (or more appropriate) speed?
Well it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.
simdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson