Re:What Comes After AGI?
by polotics on Friday November 20, 2026 @01:40AM
(#50570967)
Ok so the whole thing hangs on that notion that optimal reward maximization is optimal intelligence. The actual decision on what reward is worth, who decides on value, is conveniently taken out of the equation.
This here is hilarious: https://www.hutter1.net/ai/uaibook2.htm
$$\text{AIXI} \quad a_k := \arg\max_{a_k} \sum_{o_k r_k} \dots \max_{a_m} \sum_{o_m r_m} [r_k + \dots + r_m] \sum_{q : U(q, a_1..a_m) = o_1 r_1..o_m r_m} 2^{-\ell(q)}$$