https://www.linkedin.com/posts/emollick_kind-of-a-big-deal-a...
"... a new paper shows GPT-4 simulates people well enough to replicate social science experiments with high accuracy.
Note this is done by having the AI prompted to respond to survey questions as a person given random demographic characteristics & surveying thousands of "AI people," and works for studies published after the knowledge cut-off of the AI models."
A couple other posts along similar lines:
https://www.linkedin.com/posts/emollick_this-paper-suggests-...
"... LLMs automatically generate scientific hypotheses, and then test those hypotheses with simulated AI human agents.
https://www.linkedin.com/posts/emollick_formula-for-neat-ai-...
"Applying Asch's conformity experiment to LLMs: they tend to conform with the majority opinion, especially when they are "uncertain." Having a devil's advocate mitigates this effect, just as it does with people."
I have a hunch that, through sampling many AI "opinions," you can arrive at something like the wisdom of the crowd, but again, it's hard to validate.
> I have a hunch that, through sampling many AI "opinions," you can arrive at something like the wisdom of the crowd, but again, it's hard to validate.
That's what an AI model already is.
Let's say you had 10 temperature sensors on a mountain and you logged their data at time T.
If you take the average of those 10 readings, you get a 'wisdom of the crowds' from the temperature sensors, which you can model as an avg + std of your 10 real measurements.
You can then sample 10 new points from the normal distribution defined by that avg + std. Cool for generating new similar data, but it doesn't really tell you anything you didn't already know.
Trying to get 'wisdom of crowds' through repeated querying of the AI model is equivalent to sampling 10 new points at random from your distribution. You'll get values that are like your original distribution of true values (w/ some outliers) but there's probably a better way to get at what you're looking to extract from the model.
You only have to look at opinions about covid policies to realize you won't get a good representation because opinions will be deemed "misinformation" by the powers that are vested in that being the case. Increasingly, criticism of government policy can be conflated with some sort of crime that is absolutely up for interpretation to some government institution so people self censor, companies censor just in casa and the Overton window gets narrower.
LLMs are awesome but they will only represent what they're trained on and what they're trained on only represents what's allowed to be in the mainstream discourse.
I don't think this is a description of LLM censorship though, especially in light of the fact that many LLMs are fine-tuned for the explicit purpose of censoring responses otherwise generatable by the model, Contrasting uncensored models with censored ones yields objectively uncensored results.
<< “enlightening productivity and business scenarios” it will be an engine for simply manufacturing support for a pre-selected option.
In a sense, this is what training employees is all about. You want to get them ready for various possible scenarios. For recurring tasks that do require some human input, it does not seem that far fetched.
<< produce “convincing interactions"
This is the interesting part. Is convincing a bad thing if it does what user would be expected to see?
There's nothing unique about this tool in that regard though. Pretty much anything can be mis-used in that way - spreadsheets, graphics/visualizations, statistical models, etc. etc. Whether tools are actually used to support better decision making, or simply to support pre-selected decisions, is more about the culture of the organization and the mind-set of its leaders.
Sure, it’s just part of an arms race where having a new thing with a hot selling pitch to cover that up and put a layer of buzzwords on top of it helps sell the results to audiences who have started to see through the existing ways of doing that.
I think the issue is the human tendency to just rubber stamp whatever result is given. Not that long ago, few questioned the result of a study and now there won't even be underlying data to go back to see if someone made an error. Naturally, this would suggest that we will start seeing a lot of bad decisions, because human operators did not stop and think whether the response made sense.
That said, I am not sure what can be done about it.
While it may, as it says, produce “convincing interactions”, there is no basis at all peesented for believing it produces an accurate model of human behavior, so using it to “understand human behavior” is at best willful self-deception, and probably, with a little effort at tweaking inputs to produce the desired results, most often when used by someone who presents it as “enlightening productivity and business scenarios” it will be an engine for simply manufacturing support for a pre-selected option.
It is certainly easier and cheaper than exploring actual human interactions to understand human behavior, but then so is just using a magic 8-ball, which may be less convincing, but for all the evidence supporting this is just as accurate.