With so many weights, it practically encodes a massive Internet text database.
However, if the output needs to be curated and edited by humans, the scale and automation is gone - we just get a different manual process, with a modest improvement to speed at cost of some decline in quality, and that's not very impactful.
Google at this point favours long form content for many search intents. Being able to generate thousands of these pages in one-click is a real problem. Not just because of popular topics e.g. "covid-19 symptoms" but more so for the long tail e.g. "should I drink coffee to cure covid-19".
It may be that Google's algorithms don't care at all how human-like the text is, or that their own recognition algorithm/NN (whatever they use) isn't fooled. Even if it is affected, Google has the money and corpus to build its own competing NN to recognize GPT-3 text.
That said, there might be a different threat to Google. GPT-3 seems really useful as a search engine of sorts (with the first answer implementing the 'I'm Feeling Lucky' button). Tune it for a query syntax, and for getting the 'top X' results somehow, then we just need the web corpus and a basic filter over the results. We could have a very interesting Google competitor.
More than cherry-picking, there's the Eliza Effect - it's pretty easy to make people think generated text is intelligent. That text can seem intelligent for a while isn't necessarily impressive at all.
Makes me worry about my own reading comprehension, but I think what happened was that since it was posted on HN and got upvoted a lot, I simply assumed that anything that I didn't understand was not the writer's fault, but mine.
For instance, it was unclear from the post what the bitcoinforum experiment was about, but I just dismissed it as me not being attentive enough while reading.
At one point GPT-3 writes: "The forum also has many people I don’t like. I expect them to be disproportionately excited by the possibility of having a new poster that appears to be intelligent and relevant." Why would people he doesn't like be paricularly excited about a new intelligent poster? Again I just assumed that I missed the author's point, not that it was nonsensical.
Twice it refers to tables or screenshots that are not included, but it seemed like an innocent mistake. "When I post to the forum as myself, people frequently mention that they think I must be a bot to be able to post so quickly" seemed like another simple mistake, meaning to say that when he posted as GPT-3, people thought he was being too quick.
This is like a written Rorschach test, when I'm convinced that what I'm reading must make sense, then I'll guess at the author's intent and force it to make sense, forgiving a lot of mistakes or inconsistencies.
Bots offering idiocy and idiocy generally has done lots of damage. But by idiocy here I would quite carefully calculated cleverly polarized positions and I don't think just bot-rot would be enough (to maybe coin a phrase).
It’s cool, but it looked like very basic stuff - the type of UI that is very easy to create in a few minutes. (And really with what was setup behind the scenes - maybe just as fast to just write the code.)
The hard part about software development is not those bits which are common, but the parts that are unique to our specific solution.
Search terms tweaked for your unique interests, and not a commercial entity's, for example.
Is reddit gold really that valuable?
> famous
Surely there are easier ways.
> really useful
We already have enough 2020 reddit commenters regurgitating 2010 hn threads regurgitating 2000 slashdot threads, thanks.
This will accelerate development. Is the current version there? Probably not. But GPT-4 might, and would then accelerate the development of future versions.
Even though this is not "magic", it sounds like it will turn into a practically usable and extremely valuable tool soon.
However I like spirit of optimism and first looks at encouraging and very promising results.
Exciting times!
Honestly not that impressive since you can get comparable results with a series of regex rules given that there are limited ways to describe your intent e.g. "create a button of colour <colour> at the <location of button>"
I believe the hype is that people think they can replace the designer by "just telling the computer" what they want. I don't believe that will work, as they already have trouble telling a human what they want, and a computer won't really know what to do with "I want it to kind of feel like it's from that movie with the blue people that Cameron did, you know?"
In my experience, people have a hard time writing their ideas about designs & features down, because they don't know what they want. They want to talk about it abstractly with somebody who has a better understanding of the field so that person can help them develop the idea. I don't think ML will cover that part any time soon.
From an academic standpoint, writing is part of the thinking process. If you haven't written it down, you haven't fully thought it through. If it feels difficult, that's probably because your understanding isn't as complete as you thought it was.
From a software development standpoint, implementing something is part of the thinking process. Ever notice how the requirements have a tendency to break as soon as you actually try to implement them? If a spot seems difficult it just means you hadn't really figured it out yet.
I 100% agree. I noticed a giant shift in tasks when I made one client write tickets instead of making phone calls. Writing it down forces you to think it through.
And I agree about software development as well, yes. Though I think it's even rare to have somebody describe all the features they want unless it's an experienced software developer who basically writes a textual representation of the application.
But for most PMs (that I've worked with at least), they have vague ideas about what they want, and bringing them into focus is a back and forth with developers and designers. I don't see them getting anywhere with an NLP automaton, but maybe with an Eliza-style system: "Give me a big yellow button saying 'Sign up'" - "Why do you want a big yellow button saying 'Sign up'?" - "You're right, that's too on the nose... give me a link saying 'Sign up'"...
@balajis being generated by GPT-3 would make a lot of sense, though.
https://twitter.com/jsngr/status/1284511080715362304
Granted, it seems like there was a lot of behind the scenes work to make that happen.
It's qualitatively different than GPT-2. I was on a discord with someone that has access to it and a bunch of us were throwing ideas out for prompts. One of them was to provide an anonymized bio of someone and see if it could guess who it was. The format was 'this person...they..and then they...\nQ: Who is this person?\nA: '
At the first pass it didn't guess correctly. But we erased its response and tried again and it got the answer correct. We then asked it to screenwrite some porn and tell jokes. Yes there were some misses, but it got things right so frequently that you can start to see the future.
Having all of this capability in one package is pretty remarkable and nothing has approached it to date.
"Text generation" undersells it a little bit. What are humans except "text generation" machines? Language is the stuff of reason. GPT-3 has demonstrated capabilities that we believed were exclusive to humanity --- humor, logic, sarcasm, cultural references --- in an automatic and generalizable way. It's far more than a "text generation" system. I look forward to seeing what GPT-4 and GPT-5 can do. I suspect we're all going to be amazed by what we get when we continue down this path of simple scaling (and sparse-ification) of transformer architectures trained on (basically) the whole internet.
The ability to grow and choose our own direction: to choose what our goals are, curiosity, self-awareness, desire. To imply that GPT-3 is anything close to strong AI is kind of ridiculous.
I predict within a few years, the descendants of GPT-3 will use very different fundamental units for processing that differ greatly from the current state-of-the-art (i.e. they won't use BPEs and their ilk anymore, except for final output) and will be far more powerful as a result.
I do agree with you. We, as somewhat intelligent beings, do not base our thinking on words or language AFAIK, even though it's our best way to convey ideas to others. And we learn through experience, way faster than GPT-3 does, with fewer shots. It looks like the attention mechanisms are what made these models actually start to understand things... But those attention mechanisms are still very raw and mainly designed to be easy to execute on current hardware, I wonder how fast will we refine that. Finally it looks like, once trained, these models don't learn when we use them. It definitely doesn't learn through experience and that's a major limitation on how intelligent it can be.
I think sentience like most things is a spectrum, so I'm not really sure what you mean by sentient, but I would argue that for most people the bar for sentience is much higher than text prediction. The Chinese room is only one aspect of our minds, and we don't even know what consciousness is.
And to be fair, reasonable people stake out positions on both sides of this debate: I'm not claiming that the alternative proposition is somehow unreasonable. It's a legitimate subject of scholarly disagreement.
Nevertheless, I'm still firm on language. Why? Because all complexity is ultimately about symbolic manipulation of terms representing the process of manipulation itself. ("Godel, Escher, Bach" is a fantastic exploration of this concept.) How can you manipulate concepts without assigning terms to their parts? That's what language is.
The question I like to ask is this: are there any ideas that you cannot express using language? No? Then how is thought distinct from language?
Yes, people (myself included) experience a "tip of the tongue" experience where you feel like you have an idea you can't just yet express. But maybe this experience is what reason feels like. Why should idea formation take only one "clock cycle" in the brain? Why should we be unaware of the process?
I think this feeling of having an idea yet being unable to formulate it is just the neural equivalent of a CPU pipeline stall. It's not evidence that we can have ideas without language: it's evidence that ideas sometime take a little while to gel.
I think as highly social beings we often annotate all of our thoughts with the language we could use to communicate them, which could lead us to believe that the thoughts are indistinguishable from the language, but that conclusion seems like an error to me. I’ve also heard some people talk about how they are “visual” or “geometric” thinkers and sometimes think in terms of images and structures without words.
Not sure there's one I can communicate to you, but I'm perfectly capable of forgetting the word for something and still knowing unambiguously yet wordlessly what it is, that's an experience.
Catching a ball? Running? Experiencing emotions from wordless music? Viewing scenery? Engaging with a computer game? How are they not conscious experiences?
To me this indicates a very narrow view of consciousness. Consider for a moment the quiet consciousness of the cerebellum for example.
I like the way David F. Wallace put it: 'Both flesh and not'. There's an astounding amount of consciousness that is not bound by language. One can even argue that language might hinder those forms of consciousness from even arising.
What is the role of the body in consciousness, then?
> only context it has is its prompt
The only real context is its latent representation of the prompt, there's nothing fundamentally limiting visual, auditory, symbolic, and mixed prompts as long as they map to a common latent space and the generator is trained on it.
Text generation doesn't chop wood, optimize speedruns, build machinery or win 100-metre dashes.
Text may be involved in training for these things, but to say that doing them is text generation would be like saying that... since compiling code and running AlphaZero both generates bits, AlphaZero is a compiler.
GPT-3 is objectively a step forward in the field of AI text-generation, but the current hype on VC Twitter misrepresents the model's current capabilities. GPT-3 isn't magic.