Sounds like what you're talking about is decision theory. Or, when mixed with learning: reinforcement learning. The monte carlo AIXI approximation would be an example. But, it really only works well on small toy problems (I.e. problems where the agent has only a small number of available actions it can perform).