How? This is absolutely impossible. You can't run trials. Yes, we can analyze the effects of policies (which we should certainly be doing more of) but these results can also take decades. We can only do a singular experiment and so our sample is biased to all hell. There is no objective truth here, only results of a single experiment.
It is possible to use GPT-3 to impersonate someone. You give it a profile and it assumes the personality, then speaks from that point of view replicating its biases with uncanny ability. So what they did was to create a simulated poll using GPT-3 by aligning the model to personality profiles distributed identically to the real population.
The result? GPT-3 can predict how an idea would poll, how a population would react to a situation. So you can simulate various policies without actually having to test in reality. You can do MCMC like AlphaGO and see ten moves ahead.
It could probably be used to simulate the stock market and economy as well. Just train the model on the "digital exhaust" of companies to predict their future behaviour in aggregate.
This looks to be one of the most astounding uses of large language models. It would be of immediate interest to politicians, investors, advertisers, music and movie produces or any influencer.
It gives me heavy 'Matrix' and 'Psychohistory' vibes.
https://jack-clark.net/2022/10/11/import-ai-305-gpt3-can-sim...
The existence of adversarial attacks shows that most neural networks have pretty bad worst-case performance. Thus sticking GPT-3 into alpha-beta or MCTS could just as easily give you an ungeneralizable optimum, because optimizers are by nature intended to find extreme responses. Call it a Campbell's law for neural nets.
The actual AlphaZero nets are probably more robust because they were themselves trained by MCTS, although they still don't generalize very well out-of-sample: IIRC AlphaZero is not a very strong Fischer Random player.
Policy making is not about truth, it is about values and priorities.