greens

355 laptop years seems very low for training GPT-3

It's a paradox of the "look I proved AGI is impossible" papers.

A single consumer GPU for ~hours

This is a common misconception. GPT-3 was trained using a 300B token (~300gb) subset of common-crawl and friends. The model is larger than the dataset.

That's a great point.

You could argue despite the huge number of physical degrees of freedom, the operations on DNA can be reduced to copy, repair, express, suppress. On the other hand, there's still a ton of intrinsic complexity in storing a huge amount of data, and yeah some nucleotides are totally essential.

The other thing a wonder about: sure, maintaining a proteome is hugely complicated, but how much of this complexity goes into maintaining homeostasis (e.g. metabolism, cytoskeloton and membrane maintenance, replication,...) vs. enabling computation. Seems like silicon has the advantage here.

My 2c (apologies for the aggressive tone -- I'm just excited about AGI):

That's a very very weak upper bound on how much hardware it takes. I think it's not all that different from emulating a Nintendo64 with a quantum simulation of the hardware.

For complex systems to work (not to mention evolve), they need to be robust to small perturbations -- there's no way the computation the brain is doing is sensitive to the details of particular atoms. There has to be redundancy, modularity, etc. These things aren't human inventions so much as they are the only way to meaningfully move in a 2^|giant-number| state-space.

Ok, sorry for assuming bad faith and ranting.

Still, I think it's pretty hard to compare engineering achievements like this. Is it harder to make a fission bomb in 6 years with 1940s technology, or a COVID vaccine in 1 year with 2020s technology?

Also, I seems like we could be having this conversation in 1940 about an american battleship program.

I can't believe that you're talking about the mass-murder of civilians like it's some sort of triumph, especially since you could have made the exact same point citing the trinity test. And it's so weird that you use the term "toxic environment" right after that.

IMO AI progress in the last 10 years has drastically exceeded reasonable expectations.

You can buy a helicopter any time you want.

∫0->∞ x*e^-tx dx = -∂_t ∫0->∞ e^-tx dx = -∂_t (1/t)∫0->∞ e^-u du = -∂_t (1/t) = 1/t^2

L^1's derivative is a perfectly good function, it's not defined or continuous at 0, but whatever... it's for the same reason that the median handles even-sized sets in a special way.

0 = d/ds \sum_i | x_i - s | = \sum_i sign(x_i - s)

which is true precisely when x_i <= s for half the elements and x_i >= s for the other half (taking sign(0) = 0, sign(+x) = 1, sign(-x) = -1).

For gravity and EM fields the inverse square law is a consequence of a massless force carrier particle and Lorentz invariance. Quantum Field Theory in a Nutshell by Zee talks about this early on.