john_horton

This might be of interest: https://www.expectedparrot.com/content/johnjhorton/grant-let... It uses a collection of models to extract the text from a handwritten letter by US Grant. Probably overkill for something nicely printed.

Hey thanks for noticing - here's the MIT licensed library it's based on: https://github.com/expectedparrot/edsl

thanks! B/c it got some positive reaction here, I did a little thread on how you can turn this flow into an API: https://x.com/johnjhorton/status/1823672992624242895

Very cool - I've been working on an open source python package that lets you do some similar things (https://github.com/expectedparrot/edsl).

Here's an example of the Enron email demo using the edsl syntax/package & a few different LLMs: https://www.expectedparrot.com/content/6607caa1-efc5-439f-85...

Link to the package: https://github.com/expectedparrot/edsl

Got it - so it is the "performativity critique" - the idea that the LLM "knows" economic theories and responds in accordance with those theories. I don't think that's very likely because a) econ writing is presumably a tiny, tiny fraction of the corpus and (b) it would imply an amazing degree of transfer learning e.g., it would know to apply "status quo bias" (because it ready the papers) to new scenarios. But as the paper makes clear, you can't use it to "confirm" theories but rather use it like economists use other models - to explore behavior and generate testable predictions cheaply that you can go test with actual humans in realistic scenarios. The last experiment in the paper is from an experiment in a working paper of mine. There's no way the LLM knows this result, but if I had reverse the temporal order (create the scenario w/ the LLM, then run the experiment), it could have guided what to look at. That's likely what's scientifically useful. Anyway, thanks for engaging.

I'm sorry I don't follow - is your claim, that, say, an AI agent exhibiting status quo bias in responding to decision scenarios (e.g., a preference for options posed as the status quo relative to a neutral framing - Figure 3) that the reason this happens, empirically, is because the LLM has been trained on text describing status quo bias? E.g., like if an apple fell to the ground in an game, it was because the physics engine had been programmed w/ laws of gravity?

Are you referring to this: "What kinds of experiments are likely to work well? Given current capabilities, games with complex instructions are not presently likely to work well, but with more advanced LLMs on the horizon, this is likely to change. I should also note that research questions like what is “the effect of x on y” are likely to work much better than questions like “what is the level of x?.” Consider that in my Kahneman et al. (1986) example, I can create AI “socialists” who are not too keen on the price system generally. If I polled them about who they want for president, there is no reason to think it would generalize to the population at large. But if my research question was “what is the effect of the size of the price increase on moral judgments” I might get be able to make progress. That being said, it might be possible to create agents with the correct “weights” to get not just qualitative results but also quantitatively accurate results. I did not try, but one could imagine choosing population shares for the Charness and Rabin (2002) “types” to match moments with reality, then using that population for other scenarios." --- To clarify, this about what research questions are likely to work well here, not what questions posed to LLMs will work well.

It is so implausible that the training process that creates LLMs might learn features of human behavior that could then be uncovered via experimentation? I showed, empirically, that one can replicate several findings in behavioral economics with AI agents. Perhaps the model "knows" how to behave from these papers, but I think the more plausible interpretation is that it learned about human preferences (against price gouging, status quo bias, & so on) from its training. As such, it seems quite likely that there are other latent behaviors captured by LLMs and yet to be discovered.

that's a great suggestion - thanks!

yeah - so I think this is worth exploring. Given how many tokens you can jam in the prompt even w/ GPT3, I think could do some pretty complex game play, at least compared to what is typical in the lab e.g., I think could easily have it remember how 100 or so other agents behaved in some kind of public goods game.

oh hey - it's my paper! If anyone is interested in exploring these ideas, feel free to get in touch (@johnjhorton, https://www.john-joseph-horton.com/). FWIW - I think it would be really neat to build a Python library w/ some tools for constructing & running experiments of different kinds. I think the paper only scratches surface of what's possible (esp. once GPT4-ish has an API).

I'm one of the authors of the original paper the article is based on. Here's a link to the actual research: https://www.john-joseph-horton.com/papers/schumpeter.pdf

Yeah - I think this is just inherent in the transmission of academic work to a broader audience. Nuance and caveats get stripped away, in part because the journalists themselves aren't capable of assessing these features.

It wasn't dishonest - they didn't have the data for those multi-site employers b/c of how Washington state keeps records. The authors are upfront about this and try to address the limitation by adding a a survey component---they talked to these employers to see if their responses to the MW were different.