News for hackers. Stuff that matters.
Re:Training great LLMs from ground zero by shrubble on Thursday March 07, 2024 @12:59AM (#39625501)
Have you checked if there is a faster way to parse your JSON? 3Gbytes/hour to load a file seems slow on today's CPUs...
Re:Training great LLMs from ground zero by flybarrel on Thursday March 07, 2024 @10:52AM (#39630533)
What would be an ideal (or more appropriate) speed?
Re:Training great LLMs from ground zero by shrubble on Thursday March 07, 2024 @04:44PM (#39634920)
Well it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.
Re:Training great LLMs from ground zero by mc10 on Thursday March 07, 2024 @09:43PM (#39637254)
simdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson