News for hackers. Stuff that matters.
Re:How GPT3 Works – Visualizations and Animations by ben_w on Monday July 27, 2020 @04:10PM (#23968903)
On the one hand, yes, you should prepare.

On the other hand, when it’s good enough to replace us, it’s also good enough to replace basically any job where you transform a written request into some written output, e.g. law, politics, pharmacology, hedge fund management, and writing books.

I have no idea how to prepare, only that I should.

(Edit: what makes us redundant may well not be in the GPT family, but I do expect some form of AGI to be good-enough in 20 years).

Re:How GPT3 Works – Visualizations and Animations by nojvek on Tuesday July 28, 2020 @01:34AM (#23972603)
There’s a good book called “Rebooting AI” that does some fundamental analysis about current state of deep learning and its applications.

The biggest problem with GPT or any massive neural net is explainability. When it doesn’t do the correct thing, no one quite knows why. GPT makes all sorts of silly mistakes.

The human brain, albeit being a form of a neural net, can do some very deep symbolic reasoning about things. Artificial Neural nets just don’t to that (Yet). We haven’t figured out that not have I seen a system that is close. We haven’t got generic neural nets that can perform arithmetic operations to arbitrary precision. For computers to learn proper language, they have to embed themselves into the world for years like children do and learn the relationship of objects in the world.

So if I were a fake comment house, I’d worry about GPT. Not so much if I were a programmer or a lawyer. We do some very deep symbolic thinking to produce our work. If computers are able to replace us, they can probably replace a large part of humanity. At which point we have way bigger problems to worry about.

Symbolic reasoning is a very hard problem to crack. Something like “how old was Obama’s 2nd child when the US hit 4 digit deaths due to covid-19?”. Answering that question not only requires context like “4 digit” means 1000, it requires a bunch of lookups and ability to break a big problem into smaller problems.

Siri/Ok Google/Alexa/Cortana/GPT3 - all of them fail.

They can’t even answer “Find fast food restaurants around me that aren’t mcdonalds”.

Re:How GPT3 Works – Visualizations and Animations by phist_mcgee on Tuesday July 28, 2020 @02:50AM (#23973017)
Great comment with fantastic context and examples!