> are billion dollar industries

When speaking of billion dollar investments, a billion dollar industry is not substantial. Google and Facebook's industries are advertising, at $600bn/year. Amazon's industry is retail, at $25tn/year.

What's opened up by the GPT-3 and its prompt-programming abilities is services, without qualification. That's $50tn/year, and capturing some tiny percentage of it is what's needed to make a billion-dollar investment worthwhile.

That said, I admit this isn't the mindset most people take when they read 'substantial'.

e: I changed the wording from 'substantial' to 'transformative', thanks!

GPT-3 lets you create rigged demos to do lots of tasks but so far it's not reliable enough to do anything in production. It seems unlikely to get there using output based on random word selection. Nobody is even talking about error rates yet.

The best applications are probably when error rates don't matter because a human is just going to use it for inspiration.

I might have missed the business plan behind monetising GPT3 here. Can you elaborate on why you think prompt programming will successfully take a cut from services?

Prompt-programming is a standard features of all LMs. What differentiates GPT3 is not this application but the quality of the output. NLP companies such as chatbot providers and specialised search (patents, legal assistants, tenders) have been using domain-specific LMs for years.

You both agree I think. He's not saying that GPT-3 invented the revolutionary ability of prompt programming, but that prompt programming allows GPT-3 to be applied to arbitrary contexts (from programming to providing legal advice to generating fiction). That amazing generality and high quality allow it to be applicable to most services.

So it's taking some slice of the $50tn pie.

Yeah this is a bizarre statement considering that GPT-3 is definitely not any of those things really. GPT-3 is far, far too computationally expensive to have value in industry. A linear CRF is more useful than most NN approaches in industry right now, just simply because in many circumstances you want to have something that you can apply to a few billion documents and get the result within a few hours, then tweak a few things and repeat if you like. These simple models also have the ability to be predictable as well. Some transformer or lstm methods can be useful in industry, but it really depends on the application. I certainly would not be using GPT-like systems for much in industry, other than gimmicks for marketing. GPT-3 is useful for academia - not industry.

Hm? GPT-3 is relatively cheap to inference from, at least compared to the cost of training. You can load all the params onto a single TPU, actually. (A TPU can allocate up to 300GB on its CPU without OOM'ing.)

AI dungeon is also powered by GPT-3, and it's quite snappy. I'm not sure why GPT-3 is seen as computationally expensive, but it seems workable.

Only the premium-exclusive version of the AI model, named Dragon released last month.

Premium here means ten dollar a month, I think.

GPT-3 is not that expensive. Estimating from the paper, to train the model, the GPU hardware costs were a few million dollars, and the electricity costs were probably under 100k. This is totally feasible for many companies today, especially if the hardware is a fixed cost and can be reused for training multiple models.

And as mentioned elsewhere, inference for a trained model is much, much cheaper.

Is sentiment analysis really that good already?

Every time I’ve looked at the start of the art in sentiment analysis, it seems to be suffering from the same issue that bag-of-words has with modifiers like “not”. Or is that more a theoretical problem than a practical one?

I appreciate this is a rapidly moving field, so my knowledge could easily be out of date.

Are modifiers an actual issue for many applications?

"This isn't a terrible horrible restaurant that nobody should ever go to" seems like 1) it doesn't mean it's actually a good restaurant either 2) the writer might be joking and sarcastic and 3) this will be very rare in actual reviews.

Put another way, certain modifiers contextually go with certain words and sentiments, so why shouldn't state of the art systems lean on that fact, notwithstanding the strict application of grammar?

There was a story that a language lecturer had just explained how double-negatives were a sometimes a positive and sometimes an emphasised negative, and that likewise some languages used a double-positive to mean an negative. He claimed that English was not such a language, using double-positives only for emphasis, to which one of the students said “yeah yeah”.

An MIT linguistics professor was lecturing his class the other day. "In English," he said, "a double negative forms a positive. However, in some languages, such as Russian, a double negative remains a negative. But there isn't a single language, not one, in which a double positive can express a negative."

A voice from the back of the room piped up, "Yeah, right."

(https://www.ling.upenn.edu/~beatrice/humor/double-positive.h...)

To be fair, "yeah, right" is a sarcastic statement and linguistically the two words do not scope eachother so the positive statement is produced at the pragmatic level, double negatives are syntactico-semantic.

The issue you describe is typically called "Valence shifting" in this specific case "negation processing". It is of course a difficult problem to capture word-level sentiment and emotions but recent techniques in academic work obtain decent results.

However, industry typically relies on sentence- or document-level sentiment in, for instance, customer reviews with systems obtaining 80-90 F1-score which is very good. Often in e-commerce, aspect-based sentiment analysis is used in which a qualifying sentiment is attached to a target aspect, e.g. from a phone review systems extract: battery: large > positive; screen: dim > positive. You might have seen these types of reports in aggregate on review our e-commerce sites yourself.

It is however an ongoing field of research to process the scope of negation and uncertainty, but the field is making strides. State-of-the-art attention-based models obtain good scores on benchmark fine-grained sentiment analysis datasets such as the GoodFor/BadFor and MPQA2.0 of around 70% F1score [1]. This performance is nearly enough for commercial systems, depending on how you employ them.

1. https://link.springer.com/article/10.1186/s13673-019-0196-3

I think he means “ obvious, immediate, substantial economic value” to non technical people. It take little effort to imagine how to monetize it even for regular folks.

For many GPT1-3 is the first exposure to language modelling technology, which is great, but language modelling and pretraining is already widely used in nearly every NLP task even before GPT1. Any NLP engineer has used this, so it is kind of weird to claim large-scale pretrained LMs are revolutionised with GPT3.

Don't get me wrong the hype is largely deserved because of the performance and engineering/research/funding effort required. Plus cool demos and media marketing from OpenAI helps a lot in spreading awareness.

OpenAI has definitely revolutionised the marketing for language models, no doubt. Let's wait and see if they manage to do the same for the economic valorisation.

really? I am very impressed by GPT-3 but I still don't see any way to make money "obviously" out of it.

Maybe it can be an adjuvant to human in some tasks but then so could existing technologies too, I guess?

Do you want a computer that can reliably understand you and give you the best possible answer 99% of the time?

I think most people would go: yes! How much does it cost?

Oh, I thought we were talking about GPT-3

If you're running a search engine maybe.

But everyday people are used to getting 80% answers by search engines, I don't think many would pay for something that is "like google, but a bit better".

This seems the be current issue for many things that we used to pay for (dictionaries, encyclopaedias, newspapers, etc), and I'm not sure this would be different.

> "like google, but a bit better"

For simple queries like “who was the president of country X in YYYY” it’s probably just a bit better (if cached, of course, Google search is wicked fast).

But for more complex queries, Google is still remarkably dumb. Or downright insolent, ignoring my verbatim selection or quoted terms.

I’d pay good money for scarily smart search and a “grep for the web” service, that included JSON, CSS, JavaScript, comments, whatever. A toggle button for dumb/smart search

I would easily pay money for something that is like Google, but a bit better.

I wont

google's majority revenue is from search. So bit better than google, particularly if it is integrated into bing (as microsoft has invested in openAI) , and allows bing to capture market share from google, would be really lucrative

On the other hand, do you want a computer that gives you less accurate answers than more specialised tools but can spin them into something that looks like an essay? is less obviously monetisable than, say, already commercially available services like Alexa which incorporate NLP but don't rely exclusively on it.

GPT-3 is not capable of what you are describing.

I think regular folks are already pretty clued into how other things like text mining and sentiment determination have big markets considering how strong the backlash against tech politically is right now. The everyday public and by extension politicians seem reasonably capable of imagining how data can be monetized for ads, and they dont seem too happy about it.

Really? I have the opposite, not as positive impression. I hope you are right though.

I think it depends on how you define "clued in". They are aware of its existence but aren't just ignorant of how it works but outright apathetic and hostile to anything which goes against their personal narrative.

Just look at the "Google selling your data" being uncritically accepted when five minutes of thought would conclude it is the last thing Google would want (even a better search algorithim would find it hard to bootstrap on user base and comparable training time) or the casual John Yoo worthy torture of the definition of monopoly to include Goddamned Netflix when whining about FAANG monopolies. That level of generalization and stereotyping is like blaming the Amish for flying planes into the World Trade center because both are radical Sbraham religions.

From the "applied physics" department of John Hopkins university in Baltimore (last stronghold of the JASONs) south to Virginia and the Research Triangle Park area of North Carolina you will find people who know things about practical NLP systems that aren't in the open literature. They could tell you about it but they'd have to kill you.

Around Mumbai I know there is a crew that can really use UIMA, and there are other Indians I know who do intelligence and defense work.