I’ve been thinking along the same lines as what you seem to have done. I looked into the CommonCrawl data and also the Pile and I’m not sure they are as large as people think. We also manage some of our own hardware including GPUs. It looked to me that with maybe .5 PB decent storage and 8 high end gpus it could be possible to train a decent LLM in a few months, how does that estimate compare to your experience?