News for hackers. Stuff that matters.
Re:Google Open-Sources Trillion-Parameter AI Language Model ... by amelius on Wednesday February 17, 2021 @07:06PM (#26173647)
> it’s ridiculously parameter inefficient

How do you know that? Perhaps with this methodology you really need those 3.12TB to reach comparable performance?

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by thesz on Wednesday February 17, 2021 @07:54PM (#26174038)
I beg to disagree.

[1] provides one with a whole-data-set training method (ADMM, one of such methods). Page 8 contains figure 2(b) - accuracy of training after specified amount of time. Note that ADMM start where stochastic gradient stops.

[1] https://arxiv.org/pdf/1605.02026.pdf

At [2] I tried to apply logistic regression trained using reweighted least squares algorithm on the same Higgs boson data set. I've got the same accuracy (64%) as mentioned in the ADMM paper with much less number of coefficients - basically, just the size of input vector + 1 instead of 300 such rows of coefficients and then 300x1 affine transformation. When I added squares of inputs (for the simplest approximation of polynomial regression) and used the same reweighted iterative least squares algorithm, I've got even better accuracy (66%) for double the number of coefficients.

[2] https://github.com/thesz/higgs-logistic-regression

There's a hypothesis [3] that SGD and ADAM are best optimizers because that everyone use and report on. Rarely if ever you get anything that differ.

[3] https://parameterfree.com/2020/12/06/neural-network-maybe-ev...

So, answering your question of "how do you know" - researchers at Google cannot do IRLS (search provides IRLS only for logistic regression in Tensorflow), they cannot do Hessian-free optimization ([4], closed due lack of activity - notice the "we can't support RNN due to the WHILE loop" bonanza), etc. All due to the fact they have to use Tensorflow - it just does not support these things.

https://github.com/tensorflow/tensorflow/issues/2682

I haven't seen anything about whole-data-set optimization from Google at all. That's why I (and only me - due to standing I take and experiments I did) conclude that they do not quite care about parameter efficiency.

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by vladf on Wednesday February 17, 2021 @09:45PM (#26174851)
> they do not quite care about parameter efficiency.

Google Research is pretty big, I used to think like you did but I think it's mostly b/c DeepMind just hogs all the spotlight.

Check out PRESS [0] for example.

[0]: https://research.google/pubs/pub46141

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by thesz on Thursday February 18, 2021 @04:50AM (#26177922)
Thank you! I skimmed over the abstract and will read the paper later, it seems interesting.

But you gave me another point to support my view: PRESS uses stochastic gradient, not second-order method like IRLS.

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by vladf on Thursday February 18, 2021 @03:52PM (#26185335)
I agree, I was really only proposing PRESS for the "parameter efficiency" part of your comment. It'd be interesting to see some modern takes on IRLS. I think generally this goes against the grain of the Cheap Gradient Principle which is why you see less of it (edit: eh, I think Fisher scoring can be cast in this light).

For instance, on modern modelling problems with non-linearities and change points, it's a lot less easy to do something like IRLS in an end-to-end system, but interesting as a research direction.

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by thesz on Friday February 19, 2021 @07:46AM (#26192534)
I also agree with "Cheap Gradient Principle". I see it as a case of "width of two horse backs from Ancient Rome determine Shuttle buster width" (which is untrue but cool as a reference).

The very SGD thing was developed because it was the only way to train something like neural network with small memory and, more importantly, in reasonable time. Multiplying of training time by N (number of parameter) meant having good result in a year, not in a day.

And today we have large batch training with complex synchronization systems to speed up training even more. Which bring us closer to the whole-dataset training and, I guess, second-order optimization as well.

Re:Google Open-Sources Trillion-Parameter AI Language Model ... by buildbot on Wednesday February 17, 2021 @07:14PM (#26173719)
If you read through the paper where they present switch transformers, they train all models to the same performance, so for example t5-large, at 770M parameters[1] is just as good as this model. That’s only 1.54GB

[1] https://huggingface.co/t5-large