More on Dota 2 - Hackdot

Re:More on Dota 2 by omarforgotpwd on Thursday August 17, 2017 @03:48AM (#15034587)

I’m by no means an expert, but I’m fascinated by the idea that a neural net playing against itself can substantially outperform a supervised learning approach with a large training data set. I mean, gathering training data and making sure it’s labeled correctly and all that is a huge hassle so if you could eliminate that step or even reduce the amount or quality of training data required that should be a big win for AI, right? Especially if doing this not only makes things easier but also improves the performance of the model.

[ Parent | on HN ]

Re:More on Dota 2 by tictacttoe on Thursday August 17, 2017 @07:01AM (#15035555)

It's very cool, but I think it also requires a very specific "adversarial" problem with a well defined notion of success (winning the game). If your machine learning task is something more nuanced and harder to define, e.g. identify word synonyms, I don't see how you can get around having a training data set.

[ Parent | on HN ]

Re:More on Dota 2 by kpil on Thursday August 17, 2017 @01:33PM (#15038889)

I played with evolving algorithms playing some kind of robot war (in java) almost 20 years ago, and beside the fact that you had to be veeery patient since evolving a generation took 10 minutes at best, I realized that it's very hard to create a survival criteria that matches your intention.

I e, most robots fled and hid in the corners. I added additional critiera for hitting enemies, and the robots fled while randomly shooting bullets and hid in the corners...

No epic fights.

Hmm. Maybe I have the code somewhere...

[ Parent | on HN ]

Re:More on Dota 2 by NamTaf on Thursday August 17, 2017 @04:35AM (#15034819)

Reinforcement learning isn't a new idea - I did a Berkeley-based edX course on it a few years ago now and it was not state-of-the-art to my knowledge. That had no deep aspect to it, we just generated a reinfrocement algorithm that utilised a good measure of performance (specifically, it was pacman and the score value is pretty good at that) and changed a few algorithm weighting variables at each iteration.

My understanding, from talking to a ML friend this morning, is that the latest progress is taking reinforcement learning and applying deep learning approaches (nets, etc.) to it. The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome.

The self-play really is the reinforcement side of things at work. How you take that 'score' and use it to correctly modify the input weightings - be them in a neural net, traditional algorithm, etc. - is the key.

[ Parent | on HN ]

Re:More on Dota 2 by mannykannot on Thursday August 17, 2017 @06:40AM (#15035433)

> The key becomes finding the right scoring algorithms to tweak the neural net correctly towards the desired outcome.

Does this not become something similar to supervised learning if you are scoring internal states of the game? (i.e. scoring on more than just the outcome and things that violate the rules?)

[ Parent | on HN ]

Re:More on Dota 2 by NamTaf on Thursday August 17, 2017 @07:19AM (#15035665)

I don't know what they're scoring on; they've not elaborated on that. However, when I first read into OpenAI it looked like many of their game links were to simply view the screen. To that end, it wouldn't surprise me if they're identifying things you'd otherwise visually see in the game and simply working off them. To use internal states not immediately visible to the player would be sort of disingenuous IMO.

[ Parent | on HN ]

Re:More on Dota 2 by noway421 on Thursday August 17, 2017 @06:35AM (#15035397)

Why wouldn't algorithm reach a local maxima when playing with itself, or even degrade over time by opening up to unknown attacks?

[ Parent | on HN ]

Re:More on Dota 2 by gcp on Thursday August 17, 2017 @06:40AM (#15035427)

One typically keeps pools of trained networks to combat this.

[ Parent | on HN ]

Re:More on Dota 2 by Epokhe on Thursday August 17, 2017 @09:33AM (#15036587)

it does. the bot can be defeated easily with tactics it didn't encounter while playing with itself.

[ Parent | on HN ]

Re:More on Dota 2 by pycal on Thursday August 17, 2017 @09:49AM (#15036718)

Just a guess, but wouldn't this approach used in isolation result in a high potential for getting stuck at local optima? It seems to me that the success of this experiment is the result of interaction with players in different MMR brackets.

[ Parent | on HN ]

Re:More on Dota 2 by Bartweiss on Thursday August 17, 2017 @12:20PM (#15038118)

Yes, it can. It's fairly common to keep a pool of learners which aren't identical for exactly this reason - either playing against prior snapshots, forking training, or lightly randomizing some components of the learner.

[ Parent | on HN ]