News for hackers. Stuff that matters.
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by buildbot on Wednesday February 17, 2021 @07:02PM (#26173622)
If I did the math right it would be 3.12TB of weights, maybe they are trying to upload it to gdrive still. (/s, probably)
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by schoen on Wednesday February 17, 2021 @07:11PM (#26173684)
They released more data than that for their Google Books n-grams datasets:

https://storage.googleapis.com/books/ngrams/books/datasetsv3...

(I don't remember exactly how much it is, but I remember that the old version was already in the terabytes.)

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by lifthrasiir on Wednesday February 17, 2021 @08:24PM (#26174284)
Another example of Google giving much data away is 50 trillion digits of pi [1], which contains about 42 TB of data (decimal and hexadecimal combined).

[1] https://storage.googleapis.com/pi50t/index.html

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by dheera on Wednesday February 17, 2021 @07:58PM (#26174078)
The Waymo open dataset is about 1TB. I don't think releasing a 3TB dataset would present a technical challenge for Google.
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by londons_explore on Thursday February 18, 2021 @05:32AM (#26178212)
Even a 3PB model would be very doable for Google...
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by pradn on Wednesday February 17, 2021 @07:58PM (#26174086)
The daily upload quota for a user is ~750 GB. It'll take a few days to upload that much data to Google Drive!
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by dekhn on Wednesday February 17, 2021 @09:00PM (#26174566)
Google Cloud Storage. The files could be dumped as tfrecord in a bucket with "requester pays". So anybody could reproduce it using the open source code, by paying for the costs incurred to move the data from GCS to the training nodes.
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by shepherdjerred on Wednesday February 17, 2021 @11:58PM (#26176032)
Weights are just numbers (probably floats?), right?

This model has 3.12TB of floats??? That's insane. How do you load that into memory for inferencing?

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by exikyut on Thursday February 18, 2021 @12:23AM (#26176229)
Use x1e.32xlarge on AWS with 3TB of RAM. Just $12,742/mo - https://calculator.aws/#/estimate?id=7428fa81192c57087ac8cdf...

Alternatively order something like the HP Z8 with 3TB RAM configured, which is only $75k - https://zworkstations.com/configurations/2040422/

It's interesting. It would take ~six years for the Z8 to break even compared to AWS, but traffic into and out of the machine would be $0, and I don't think you're running directly on the metal with AWS, so performance would probably be a bit higher. And then there's storage - I configured, uhh, 120TB of a mixture of SSDs and HDDs. I'm not even going to try and ask AWS for a comparible quote there.

I may or may not have added dual Xeon Platinum 8280s to the Z8 as well. :P

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by tjbiddle on Thursday February 18, 2021 @03:42AM (#26177445)
When you're spending that kind of money on a machine, there's no way you're paying retail price. Sales reps would give you a significant discount.

Also - think you meant 6 months, not 6 years anyhow :)

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by exikyut on Thursday February 18, 2021 @05:33AM (#26178223)
Interesting. I'm very curious... 20%? 35%?

And I did mean 6 months, woops. Didn't even notice...

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by jsnell on Thursday February 18, 2021 @12:53AM (#26176439)
> It would take ~six years for the Z8 to break even

Do you mean six months?

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by exikyut on Thursday February 18, 2021 @05:29AM (#26178193)
Oh *dear*. I definitely tripped over there, and I didn't even notice.

Yup.

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by high_byte on Thursday February 18, 2021 @06:50AM (#26178764)
Z8 sounds like fun. but I might just buy two teslas (roadster and X, or a cybertruck) and a gaming PC. :D
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by visarga on Thursday February 18, 2021 @01:00AM (#26176482)
hate to break the party but this model only loads a small part of itself in RAM when inferencing
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by exikyut on Thursday February 18, 2021 @05:31AM (#26178203)
That's a good thing. Less completely means more energy for interestingness, and less expense means more accessibility.
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by swirepe on Wednesday February 17, 2021 @07:33PM (#26173890)
(They are definitely going to exceed their storage quotas.)

I want to see how well weights for these models compress, but it will take me some time to run this code and generate some. I'm guessing they won't compress well, but I can't articulate a reason why.

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by wisty on Wednesday February 17, 2021 @07:45PM (#26173975)
If weights compress, they have low information, which would suggest they're either useless or the architecture is bad.