They are testing the AI's ability to do tactics and strategy, not motor and visual tasks. It has built in delay and needs to move around the map to gather information just like a human, so it doesn't have any significant unfair advantages.

Edit: Note that the version they sent out to the ladder had significant larger delay, significant lower APM, and didn't get any information not visible on the screen unlike the first iteration.

When Watson won Jeopardy, its lead designer thought a 5 ms reaction time was fair because one time a human anticipating the buzzer light got 2 ms.

Of course to any reasonable person a consistent 5 ms is obviously vastly superior to once in a blue moon lottery-winning luck.

Sounds like it's the same deal here, where the designers made some effort to level the interface playing field, but also left in some advantages like instantly being able to select individual units (or whatever it was the other players felt was not humanly possible to do). Probably because, at some level, they really want their system to win.

Yes, the point of the entire project was for the system to see if it can win.

If we have reached the point where to even risk a loss requires implementing artificial restraints on the AI...well...

They aren't really artificial restraints though, more like artificial advantages without them. The interface delay challenges are pretty relevant to making things fair in almost ANY competition. The physical part of the interface is an essential part of the game here.

You aren't really testing the thing you really want to test otherwise.

You could probably play tennis pretty well against Nadal if you made him wear goggles that gave him a quarter second delay in response time, but that's not really a fair test.

All other things being equal, if you match up two chess players, and one can just say or think the position instead of having to physically move the piece before they hit the clock, that advantage will accumulate over time.

I wonder if implementing tactical pause would eliminate interface advantages. Players can pause game, scroll around map and give orders to units before unpausing and watching them execute.

It will be a different game, but you can see how AI does against humans tactically and strategically.

Strategy is limited by available tactics, and the ability to execute them. Your (human) tactics and strategy would necessarily change if you and your opponent were required to use a mouse that can only move across the screen over the course of a couple seconds, or if you could only click once per second, or if the computer added noise to when/where your mouse click arrives.

Strategy and tactics are a function of constraints. If an unconstrained computer can beat a constrained human, have we really shown that its strategy and tactics are better? To do that, you'd have them to play under the same constraints. Or at least have the constrains be close enough, which is what people are debating.

However the computer can macro while in a fight which is something people can’t dI. We would miss the fight and lose the game. It’s not just a tactics game. Attention span and where your eyes are matter.

It’s so bad that below master people shouldn’t really be doing too much micro during a fight because they’ll lose out on macro. Even in pro matches you see attention span issues and sometimes avoidance of too much micro to win. Really good Micro is going to win you any battle.

I’m in diamond 2 and I definitely micro in battles. The difference between platinum 1 and diamond 3 is largely macro but it very soon becomes micro as well.

Elite players do this all the time. Serral would use his standard tactic of attacking in two places as the same time, and alpha star couldn’t figure out the best way to deal with it. When alpha star did the same thing to him, he would mitigate the smaller attack and even predict alphastar’s escape route and crush it.

The competition is not the goal - the point isn't to see "who's better", the competition with real humans is just a tool to help improve the reasoning, planning and decision making capabilities in a less artificial environment than usually. The purpose of that project is to use Starcraft as a playground to test, review and improve reinforcement learning methods so that they can be used for other needs.

Any aspect of "fairness" is relevant if and only if it helps that goal. APM limits are important because we want the computer to choose strategies that would actually be good choices in that situation even for a human player, instead of being able to win with strategies that are powerful only because of superhuman clicking speed. On the other hand, requiring the computer to press buttons with a robot hand doesn't really facilitate improvements to the decisionmaking part, so it's irrelevant even if it would make it more fair.

Yeah. Bulk of the other comments are useless for that reason. It's almost like humans have inferiority complex against AI...

Interesting thing about AI is no matter how you spin things, the more advanced it becomes, unfairness grows proportionally.

After all, isn’t that the point of AI? To perform better at certain tasks than humans? We are visitors in the digital realm, just like when we put on a scuba suit and jump in the ocean—even the smallest fish can out-swim us.

What are you solving for here? They already limit the AI to human like speed, reaction times, and accuracy.

Then we are stuck with an inherently unfair system, as robotics is nowhere near ready for keyboard/mouse, and neuroscience is nowhere near ready for CBI.

This current modality is important, IMO, because we could potentially see neural networks performing tasks on other software, not just SC2. Imagine a neural network performing copy-editing in Word, writing code for CRUD applications, etc. Those are some mind-blowing potentials, we'd be losing out if we slowed down to work on robot hands.

some korean players were so quick at inputs you'd think they were half way to brain-computer interfaces already

Interestingly, the hardest part of AlphaStar looked like it was researching how to stop the computer winning through an overwhelming mechanical advantage.

Depends on what you're trying to test. You could just as easily argue that _not_ allowing the AI to make use of its natural APM and interface advantages is unfair, and that it should be allowed to play without restrictions in order to demonstrate its full potential.

AlphaGo had an "unfair" advantage in its games against Lee Sedol in that it was able to "think" far faster than you could reasonably expect a human to, and could therefore evaluate many, many more moves than a human player possibly could have. If you artificially limited the speed at which AlphaGo was able to "think" to match human limitations, would it have been able to win against Lee? I'd argue probably not.

Similarly, here we have an AI that's easily able to crush top SC players if it plays with an uncapped APM and no camera limitations, but only reaches grand master level when you impose artificial restrictions that force it to behave more like a human. The question of which configuration is more "fair" is kinda arbitrary; it depends on what your goal is.

If you read the paper, this is exactly what they have done, and this is why this achievement is so interesting.

Maybe we can train a better AI at turned based strategy games.

I hope Civ5 could open up its API.

Or even a turn based game, where click speed does not matter...

But reaction time in a real time game is also an interesting problem with AI. If they factor out the merely mechanical (the UI itself), reacting to a situation unfolding in real time where you don't have a lot of time to think is a nice test of AI vs human strategic and tactical thinking.

If Google could tackle Civilizations god-aweful AI next, I'd be over the moon.

Do you mean Google should write a better AI than the one in Civilization?

Interesting. But also consider this: in casual strategy videogames -- actually, strike "strategy" and just consider videogames -- most players don't want a really hard opponent. A computer opponent that is really very hard to beat is not what we want, because that'd be frustrating and many of us play videogames (yes, even strategy games!) to unwind; we want the illusion of challenge, an opponent that is challenging to beat but within the possibilities of every person who buys the game.

Which, by the way, is also the case with Starcraft II. Most people who bought it aren't tournament players. They expect a challenge, but not a really hard challenge. Game difficulty is all about perception ;)

PS: I shamefully confess I reloaded my X-COM (DOS!) game every time I lost one of the soldiers I was emotionally attached to. I don't like losing! :P

In strategy games like Civilization the "preferred" difficulty level is that hard/easy to beat because it gets extra resources. It would be preferrable to have the same difficulty level through opponents that play better/smarter while having the same game mechanic consequences as players if they make the same actions, but we currently can't, so they get artificial production multipliers and such.

Yes, I understand this. I guess I disagree better/smarter would be better, because in videogames what matters is the illusion of challenge, not a real challenge. So spending resources into developing a real AI for Civilization is probably not the best idea; as long as it tricks casual players into believing it's putting up a fight, that's good enough.

This approach creates a mismatch between singleplayer and multiplayer modes - this means that playing against a computer opponent rewards/requires different strategies than a human; a challenging computer opponent has more income and units but poor usage of them, while a similarly challenging human opponent has less income and units but uses them very differently; so playing against challenging computer opponents doesn't help you improve against other players but possibly is even counterproductive as you learn to adopt strategies that are bad in the other environment.

True. I don't know many videogames in which playing against the computer really helps you against human opponents.

I don't know whether a more capable non-cheating AI would help though. Not unless it specifically imitated how a (good) human opponent would play, which I guess is an additional and difficult to implement constraint.

I mostly agree with you - there is a little bit of a fine line though. I can learn from my opponents strategy playing against someone a little bit better than me. Maybe i lost because i need more or less of X? Tough to learn from when the opponent is creating the illusion of playing well, rather than playing well.

Current civ A.I.s may not be the smartest that the Civ team could do. Sid Meier once said on video that on play tests, when the A.I. would do something brilliant, players would just assume it's cheating.

I would pay a lot of money to be able to play a human level Emperor+ level AI on civ

My understanding is they've capped the AMP/EPM against a reasonably old EU/NA player.

Or connect the computer to human neurons directly.

You'd be able to save the time it takes a nerve impulse from head to hand that way. It's not much but apparently it can be significant enough that in some DotA games shorter players are disproportionately successful in the roles requiring the fastest reflexes.

Is that supposed to be a joke? DotA doesn't rely on reflexes nearly enough for that to matter.

Data doesn't lie.

When so many player play a game, even an almost irrelevant advantage can tip the scale.

Or maybe it's a sociological factor, not a phisics ones.

But the data doesn't lie.

Sure, data doesn't lie, but where is the data? I'm a pretty seasoned DotA player myself and this is the first I'm hearing of any such phenomenon. I can't think of any hero that shorter players have been more successful on than taller players, or vice versa.

22 actions each 5 seconds isn't that generous. Pro players have to manage apm too. Conserve it or spend where needed. I would say it is fair.