> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future

Ads already ruthlessly exploit these people, taking their money in exchange for garbage products. LLM might make this worse, but we have already automated displaying targeted manipulative messages via ad networks. People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots.

> People who can handle manipulative messages from ads should be mostly resistant to manipulative messaging from bots.

Not necessarily. LLMs at the model of GPT-4, and eventually above, affect the economics of advertising. Currently, there's only so much sense in optimizing your ads before it's cheaper to just spam more of it (er, "increase exposure"). LLMs will make it cheaper to optimize the ads further (i.e. get more bang for the same buck), so I expect the baseline manipulative effect of most ads to increase.

Why work on a broad message (ad) when you can afford (cpu time wise) to have a dedicated con artist target every individual in the world at the same time?

Exactly, and thanks for making this point explicitly.

It's one of the end-game consequences of the change in advertising economics I mentioned: with costs of future AI models going down, eventually companies may be able to afford to generate individually personalized and highly persuasive ads on the fly. A personal dedicated con artist for everyone, indeed.

What's more these people often don't know it's even happening.

In fact, the odds are extremely high that you, me, and everyone else here has been taken in by some kind of ad or ad-like rhetoric and internalized it as just being normal world-view.

If there exist those who haven't even been fooled in such a way, they must certainly have the self-awareness to nevertheless entertain doubts on the matter.

Yes, everyone has to one degree or another. No one is an expert in all fields, no one has perfect knowledge and so you will often be presented with things that seem perfectly reasonable but which are entirely false. Worse yet, most of the information you receive will be partly true and partly false with no easy way to tell which is which without engaging in a significant amount of legwork. The trope is the Gell-Mann Amnesia effect which you face every morning when you read a few interesting articles or listen to the daily news. The practices of active deception for sales and influence are all quite clearly laid out by Edward Bernays in his book Propaganda, even our modern concept of "influencers" was known during his day, the more exoteric framing being that people will trust and follow the advice of people that they like or think are similar to them in some way.

I have a new law: "Sufficiently large quantitative change always turns into qualitative change."

> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future

Versus the last thousand years?

Just wait until you learn about organized religion it’ll blow your mind.

> Just wait until you learn about organized religion it’ll blow your mind.

That gets super complicated, because arguably religion is less about its content, and more about the community. The hunger for connection isn't going away.

In a sense, religion is the perfect intersubjective phenomenon - the actual beliefs don't objectively matter and can be arbitrary[0], what's important is that everyone in your community subscribes to the same arbitrary set of beliefs. Traditions and proclamations are effective proxies for community cohesion.

(Arguably, national propaganda is the same story, just played on fast-forward.)

Point being, religion is much more than being smooth talked into believing nonsense. It digs roots deep into how humans form societies.

--

[0] - Mostly. Some beliefs are more sticky or self-reinforcing in the real world than others. Religious beliefs undergo natural selection too.

Yes it generalizes well beyond religion. All you need is something that distinguishes you and your group from others, something that is relatively easy to achieve but requires a sign of commitment. It can be how you speak, what you say, what you wear, what you eat, what you say you believe, anything really.

The people I'm less attracted to connect with are religious people, I actually have to force myselft to overlook the religiosity of my mother to live in harmony with her, but it does require effort I rather not do with anybody else. So maybe that's not the whole story, or maybe it's just a modern pitfall that wasn't as important in older times.

> Religious beliefs undergo natural selection too.

All memes do - that's why their name is based on genes. What's interesting is discovering clusters of memes that are self-reinforcing and are passed down consistently, without deviating too much from the original (while still allowing changes), over a long period of time. Arguably, religions built around such clusters have an easier time getting adherents and can get resurrected tens or hundreds of years after being nominally defeated or purged. So I don't think the "actual beliefs (mostly) don't objectively matter" characterization is warranted. I think they do matter a lot from the evolutionary standpoint; not so much for any of the current adherents, but for their grandchildren and later descendants.

To what extent the memetic content carries the institution versus how much an established institution contributes to propagating the content is unclear to me. There's definitely a relationship there, a complex interplay between vested interests, provided services, and people's needs and expectations. As you said, religion is much more than just a set of "nonsense" people get talked into. This makes it hard to discern the real utility of a particular nonsense for ensuring the religion survival.

One rarely discussed peculiarity about religion is that it distorts reality for both believers and non-believers (as one can see by observing how anti-religious people discuss what religion "is", what religious people believe, etc).

Possibly relevant:

LLMs confabulate not hallucinate

https://www.beren.io/2023-03-19-LLMs-confabulate-not-halluci...

Are you a bot? You just submitted the article header back at this person

“Organized religion” describes a lot of things, only a subset of which is actually organized religion.

There are plenty of examples, yes. Abstract art: a bunch of meaningless color patches in which certain viewers manage to find something "deep". Other examples are audiophiles and sommeliers. All those are well known to pay solid sums of money for something they are just imagining, to sellers who managed to exploit this effect.

But creating a new cult (let's call them all "cults") is rare. It takes a person with a unique combination of skills and talents, I bet most of us can't start a religion or succeed at selling snake oil at will. But with help of LLMs...

People who don’t know how to make sense are more likely to be exploited, than to automate their accidental exploits.

> I think any human who confuses smooth talking with making sense will be ruthlessly exploited in relatively near future

Buddy, they already exist, and they're carrying MBAs, Teaching Degrees, and working the Stock Market. Anecdotally, one of the few bona-fide psychopaths I met face to face submitted a thesis that everyone in the school system bent over backwards to say was amazing, on actual review had submitted a complete word-salad.