Re:Evolution Strategies as a Scalable Alternative to Reinfor...
by vn0m on Saturday March 25, 2017 @06:07AM
(#13954848)
I'm not an expert but for me it's seems that the proposed approach doesn't compute the value function . By optimizing directly the policy function aren't we losing some key ingredient for generalization? I mean could this algorithm be used in a fully non deterministic environment ? Like human vs machine?
[ Parent
| on HN ]
Re:Evolution Strategies as a Scalable Alternative to Reinfor...
by alfa02 on Saturday March 25, 2017 @10:23AM
(#13955766)
I am not sure at the moment. I guess the main problem of ES in a complex non-deterministic environment is that it would average over multiple local minima that occur in one generation which would result in a non-optimum solution. There are policy search methods that address this problem (e.g. VIPS https://scholar.google.com/citations?view_op=view_citation&h... , hierarchichal REPS https://scholar.google.com/citations?view_op=view_citation&h...).
Temporal credit assignment is another problem: the policy is updated after a full episode and there is no way to use the information which step was responsible for which reward. Policy search usually works well if the value function is very complex and the optimal policy is simple.
Re:Evolution Strategies as a Scalable Alternative to Reinfor...
by dimatura on Saturday March 25, 2017 @01:42PM
(#13956698)
It does not, at least not explicitly, which is a distinguishing feature of policy optimization algorithms. I don't think there's anything about this that makes it worse for non-deterministic problems. Note that the policy can still be stochastic, if desired (not sure if that's a good idea in general). A nice feature of black box policy optimization is that it's almost trivial to apply to either stochastic or deterministic problems.