News for hackers. Stuff that matters.
Re:GPT-3: Language Models Are Few-Shot Learners by nafey on Friday May 29, 2020 @12:52AM (#23346372)
What is it about ai generated texts that on skimming through it it makes sense, but if you try to slow down and understand it feels absurd and surreal.
Re:GPT-3: Language Models Are Few-Shot Learners by mannykannot on Friday May 29, 2020 @08:44AM (#23349284)
Because language is being treated as a thing complete in itself, as opposed to being related to an external world?

One of the issues in the 'Limitations' section was a difficulty with "common-sense physics", such as with the question "if I put cheese into the fridge, will it melt?"

To answer that question, you have to ask the right questions, such as "what is a fridge?" "what is a fridge for?" "What does it mean for cheese to melt?" "what is the cause of cheese melting?" Then one should consider the follow-on questions, such as "what are typical fridge temperatures?" "what are typical cheese melting points?" "what temperature is the cheese likely to be at initially?" (at which point, it helps to introduce the concept of room temperature, and note that it typically falls between the other two.) From facts such as the answers to these questions, one can deduce the probable outcome of putting cheese in a refrigerator, but none of the answers so far explicitly state it.

Is it plausible that any learning, solely from the structure of and correlations between examples of language use, could develop the sort of analytical/modeling approach that I have just outlined? Instinctively, I don't find it very plausible, but I am not very certain in that view.

Re:GPT-3: Language Models Are Few-Shot Learners by keenmaster on Saturday May 30, 2020 @12:51AM (#23358572)
Maybe there exists the following distinction. Modern language, as it's actually spoken most of the time, is like a higher level programming language. The structure of our brain, combined with our senses and the uber-simple way that we're taught as infants, is like lower level OS programming (parent points to hot food - look Johnny, it's hot! hot! - makes Johnny touch the food - food hot! - blows on the food to make it colder).

Some words like good/bad, hot/cold, and important/unimportant are underrepresented in everyday speech compared to the prevalence of the underlying concepts. That's why I'd categorize them as lower level word-concepts. This distinction, about variable levels of abstraction, might be important for true AI. Think about how many years it takes for humans to develop highly abstract cognition. That whole time our operating system is being coded. Maybe we need to approach AI in the same way.

It's not just AI that can benefit from better lower-level understanding. Seeing language in the above way, we can re-frame Ludwig Wittgenstein's philosophy and its normative implications for human communication. Our "programming" (communication) is on average too higher level. Excessively abstract instructions make it harder to decode and process in a precise and efficient manner.

Re:GPT-3: Language Models Are Few-Shot Learners by KhoomeiK on Friday May 29, 2020 @01:41AM (#23346607)
Yup, definitely different from how humans learn. A baby's speech would be almost the complete opposite, grammatically incorrect here and there but constructing a coherent line of thought for the most part.
Re:GPT-3: Language Models Are Few-Shot Learners by ragebol on Friday May 29, 2020 @04:32AM (#23347654)
My thoughts exactly. It seems a person incapable of proper grammar (like a baby) has some concept or thought it wants to express, but can't because it doesn't know the words etc.

These language models seem to know the words and the grammar etc, but lack a underlying concept they want to express.

There are systems that derive 'thought-vectors', but I'd be interested going the other way: somehow create such a 'thought-vector' and generate text to express that thought.

I don't know how to construct a 'thought-vector' of any concept though.

Re:GPT-3: Language Models Are Few-Shot Learners by MauranKilom on Friday May 29, 2020 @09:14AM (#23349562)
I have been thinking about this kind of thing too. What if there was some way to feed your condensed thoughts into such a model and it writes a paper/blog post/article?

Essentially, one should be able to use these models to "interpolate" the writing around the raw meaning/content. Typing assistance (think Grammarly) already allows you to refine finished writing to be more in line with what some language model expects, but imagine if it actually generated most of the text for you, based on small bites and chunks you throw at it.

Re:GPT-3: Language Models Are Few-Shot Learners by runawaybottle on Friday May 29, 2020 @09:48AM (#23349919)
If we get to large scale text generation like that, we are all going to have to become even better skimmers due to how the meta language will evolve.

So take your standard press release. We know about two thirds of it is just fluff. In other words, we are accepting the mass of fluff as one word in our language, it translates to ‘ignore’.

Our own language will change in that case.

Re:GPT-3: Language Models Are Few-Shot Learners by KhoomeiK on Friday May 29, 2020 @03:03PM (#23354161)
What might some valid sources of data for this be? Perhaps comparisons of Simple English Wikipedia to the standard English Wikipedia? We'd need a side-by-side comparison of condensed information and a fluffed up piece.
Re:GPT-3: Language Models Are Few-Shot Learners by md5person on Friday May 29, 2020 @04:02AM (#23347455)
I think it's because the generated text will generally follow a reasonable "structure", it's "framed" relatively well and you (as a person) will recognize those patterns quickly. There's plenty of "glue" used throughout the text. Those are all patterns which we pick up very quickly, and we're used to seeing them in "real text".

Exampels from the comment above include using things like: "A year ago", "made headlines", "Now, it's the {event}, and {name} is at it again. But this time, ...", "{name} was not impressed", "You know, I feel like, I feel like you could have ...", "I don't know if... but...".

Those are all very common in those "online celebrity magazine" type texts...

It's only when you actually read into the stuff that's in-between, you'll come to see it's pretty much a load of nonsense. But that takes a bit more time and slower reading.

Re:GPT-3: Language Models Are Few-Shot Learners by peteretep on Friday May 29, 2020 @02:22AM (#23346812)
There are some very prominent politicians whose primary mode of speech is the same