Re:Training great LLMs from ground zero
by flybarrel on Thursday March 07, 2024 @10:52AM
(#39630533)
What would be an ideal (or more appropriate) speed?
[ Parent
| on HN ]
Well it would depend on the specifics of the JSON file but eyeballing the stats at https://github.com/miloyip/nativejson-benchmark/tree/master seems to indicate that even on a 2015 MacBook the parsing proceeds using e.g. Configuru parser at several megabytes per second.
simdjson can parse JSON files at ~2.5-3GB/s: https://github.com/simdjson/simdjson