News for hackers. Stuff that matters.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by ravetcofx on Tuesday May 28, 2024 @12:55PM (#40502797)
Accessing the dataset to train from scratch will be the biggest hurdle, now a lot of the pile has had ladder pulled since GPT-4
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by GaggiX on Tuesday May 28, 2024 @01:55PM (#40503500)
https://huggingface.co/datasets/HuggingFaceFW/fineweb has 15T cleaned and deduplicated english web data tokens.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by ravetcofx on Tuesday May 28, 2024 @02:27PM (#40503867)
Holy crap, Does huggingface charge for bandwidth if you're downloading 45 terabytes??
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by drexlspivey on Tuesday May 28, 2024 @02:47PM (#40504075)
I believe they are hosting it on Cloudflare who doesn’t charge for egress
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by fragmede on Tuesday May 28, 2024 @03:02PM (#40504266)
More specifically, Cloudflare R2 doesn't charge for egress, and Cloudflare doesn't charge for egress to members in the Bandwidth Alliance which include Azure, Google Cloud, Oracle, Alibaba Cloud, and others, though critically not AWS.

They very much do charge egress fees elsewhere.

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by andersa on Wednesday May 29, 2024 @12:21AM (#40508432)
Fun trivia: downloading 45TB costs about $60, according to Cloudflare.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by verticalscaler on Wednesday May 29, 2024 @07:17AM (#40510685)
That's what Cloudflare charges. It costs them around 6 cents.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by kazanz on Wednesday May 29, 2024 @12:56PM (#40514133)
Wish I could say I'm surprised you're getting downvotes. Carrier costs are some of the lowest costs for hosting providers. Yet that fact seems to elude a majority of the community here.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by andersa on Wednesday May 29, 2024 @05:36PM (#40517311)
That's what they said it costs on their blog, not that they charge that. https://blog.cloudflare.com/aws-egregious-egress

Where are you getting 6 cents from?

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by CamperBob2 on Tuesday May 28, 2024 @01:15PM (#40503003)
Someone will come along and say "Why don't you just mirror Anna's Archive?" in 3...2...1...
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by sebzim4500 on Wednesday May 29, 2024 @05:32AM (#40510100)
I think between Anna's Archive, fineweb and as many github repos as you can scrape you can get a pretty decent dataset.

I doubt Anna's Archive would produce a good model on its own though.

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by meiraleal on Tuesday May 28, 2024 @01:16PM (#40503006)
I'm okay with paying for datasets
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by CamperBob2 on Tuesday May 28, 2024 @01:27PM (#40503150)
Depends on how the courts rule. If the copyright maximalists prevail, only the wealthiest entities will be able to afford to license a useful data set.

Paradoxically enough, this is the outcome that most "Hacker News" denizens seem to be rooting for.

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by groby_b on Tuesday May 28, 2024 @03:58PM (#40504918)
It's almost as if people believe in fairness and compensating people for their work.

Also, it's worth noting that this is only true as long as we're stuck in the "must train on the entire sum total of human output ever created" local minimum for machine learning. Given that most biological entities learn with much less data, this might well be the thing that prods ML research to using an approach that isn't "IDK, buy a few containers of GPUs, and half a DC of storage, see if that makes things better".

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by nwsm on Wednesday May 29, 2024 @11:24AM (#40512964)
> It's almost as if people believe in fairness and compensating people for their work.

Yet in this case we are talking about compensating the compilers/massagers/owners of the datasets, not the original authors from wherever the data was originally scraped.

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by wizzwizz4 on Wednesday May 29, 2024 @01:45PM (#40514670)
Copyright is hideously broken, but in theory: the owners only own it because they compensate the authors, which they only do out of an expectation of future profit (on average).

That theory's a fantasy, because extractive systems involving gatekeepers get established, but in this specific case, enforcing copyright would make things fairer for authors. There's no extractive copyright-taking gatekeeper for websites: scrapers don't get copyright, so can't re-license the material they've scraped (unless it's permissively-licensed or something).

Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by meiraleal on Tuesday May 28, 2024 @01:53PM (#40503463)
I'd still get most of my dataset from torrent but I could pay for specific things like high quality source code.
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by exe34 on Tuesday May 28, 2024 @01:15PM (#40503004)
i suppose you wouldn't be able to use it for external services, but internally, I'm sure you can find some books that fell off the back of a truck...
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by HeatrayEnjoyer on Tuesday May 28, 2024 @01:20PM (#40503065)
No reason you can't go external. GPT was trained using ebook torrent sites
Re:Reproduce GPT-2 (124M) in llm.c in 90 minutes for $20 by artninja1988 on Tuesday May 28, 2024 @02:33PM (#40503937)
OpenAI has enough money to hire lawyers to defend it until the end of time though