I’m also curious if an AI could process the screen feed quickly enough to compete in first-person shooter games. Seems like it would be difficult without extremely high end hardware for the foreseeable future?

There's already models specifically for things like identifying players in Counter-Strike 2, including which team they're on.

Someone has even rigged up a system like that to a TENS system to stimulate the nerves in their arm and hand to move the mouse in the correct direction and fire when the crosshair is over the enemy.

We are definitely already there.

They documented it on YouTube for us to see:

https://youtu.be/9alJwQG-Wbk

holy shit that's amazing.

I had students build this kind of thing in 2020 by screenshotting the game and processing it with a standard OpenCV pipeline. No GenAI needed.

Thank you for educating me. How does OpenCV work from the perspective of recognizing things in an image? Is there some kind of underlying model there that learns what a target looks like or not?

The way they did it, they were writing an aimbot. So the pipeline was:

- take a screenshot

- run massive skeletal detection on it to get the skeletons of any humanoids

- of those skeletons, pick a target closest to the player

- for that target, get coordinates of head node

- run a PID to control the cursor to where the head node is located

- move the camera one frame, repeat the whole process. If you can fit that pipeline to 16ms it can run in real time.

Wow, that's fascinating. Were they able to fit the whole thing inside the 16ms frame?

oh yeah, with little problem, especially with a GPU it's not hard at all.

It already exists.

If there's enough players, the server could segregate them in tiers. All superhuman players would eventually end up competing between themselves.