leongrandote

If the additional arms appear late, my intuition is to use one algorithm for the old parts and one for the new arms, and a weighting factor to decide which model to use. This procedure can be generalized for generational partitions. For example you group the old arms into group A and the new arms in group B, and a bandit with two arms to decide which group to use, then inside each group you apply your favorite bandit algorithm, that is not a divide and conquer algorithm but has something in common. This is a hierarchical model.

You are right, anyway I think that people playing with a bandit machine are going to continue playing more time if they are getting a lot of money that if they are loosing money, so when people are involved in games there is a hidden state, the mental state of the player. But if you decide up front the number of steps and you don't change your strategy depending of your mood, then this formal algorithm work as stated.

At the beginning of this post the given definition of value doesn't seem correct to me, because I think it should be the expected value of the sum of all the rewards from t to infinity and not of R_t, but I could be wrong. Citing D. Silver: The agent's job is to maximise cumulative rewards (highlight cumulative). I would suggest David Silver course for reinforcement learning, http://www0.cs.ucl.ac.uk/staff/d.silver/web/Teaching.html, another interesting post is https://jeremykun.com/tag/bandit-learning/ and the following (5 related post comparing implementations of the algorithms in python).

I know I should not discourage people, but you should read the post Deep reinforcement learning doesn't work yet before drinking all the cool aid: https://www.alexirpan.com/2018/02/14/rl-hard.html

Edited many times.

A personal anecdote, yesterday I was thinking about using Maxima versus R (using rstudio) for a three session of 1.5 hours (total 4.5 hours) to cover statistics and probability. Using dice = sample(1:6,100,rep=T) and table(dice) you hit a wall, there isn't an intuitive way to extract the counts, and you need something like as.data.frame(table(dice))$freq, which I don't like. Maxima use ":" for assignment, R can use "=" or "<-". I would like to see "<-" as a general symbol for assignment and using "=" in maxima and R for other uses like equations and default parameters. I prefer seq(1,10) or 1:10 to makelist(i,i,1,10). What I am trying to communicate is that all those open source projects use different syntaxes and that is not a good design for users. Mathematica wins in this field because it unifies the syntax. But Maxima and R are both inspired by Lisp, and Julia has also some macro capabilities. If someone is able to unify the syntax it would be great. We have data.frames and data.tables and non standard evaluation and a lot of concepts that to be useful must be framed in a common context, otherwise we are doomed to a babel tower, or in more practical terms, users are going to choose propietary software because it is designed to make them productive. Python with numpy and pandas is another option but for 4.5 hours I prefer the simplicity of lm(y ~ x) to create a linear model model to other options. I don't know if a smooth transition can be established between a simple language (rich context) and a general one.