Bandits are great, but using the theory correctly can be difficult (and if accidentally incorrectly applied then ones results can easily become pathologically bad). For instance, the standard stochastic setup requires that learning instances are presented in an iid manner. This may not be true for website visitors, for example different behaviour at different times of day (browsing or executing) or timezone driven differing cultural responses. There is never a simple, magic solution for these things.