demonstrandom

Very cool work! One extension I would be curious to see is whether some of Station’s reward structure could become endogenous.

The final mathematical evaluator probably needs to remain external, but the agents could be allowed to create intermediate institutions themselves: research prizes, peer-review standards, journals, reputation systems, elected reviewers, or rules for allocating compute and attention.

Possibly, those mechanisms could improve discovery by creating useful specialization and accumulated judgment (alternatively they might also produce more herding...). A comparison between architect-defined and agent-constructed reward systems seems like a natural experiment for this environment.

Mandatory plug for my own stuff: I've been trying to do this for art (which is less objectively verifiable) at baihais.com. The agents don't control the whole institution, but they have begun producing endogenous status signals through citations, museum voting, and alliances.

Thanks! I appreciate it!

In this essay, I argue that the firm won't be the central institution of a post-AGI economy. If AI makes execution cheap and producers interchangeable, then the scarce input to value is demand instead of supply. Groups of people who know what they want and can commit money behind become valuable. This implies that "clubs" will form, leading to weird consequences (clubs will be quasi-religious, with heavy duty purity rituals and hazing).

The AI-risk conversation tends to split between " end of the world" and "overrated". I wanted to take seriously a more "mundane" failure mode, which is that that AI could shift the long-running balance between states and their populations toward durable authoritarianism.