Interesting approach! I've thought about a similar method after reading about the PLATO platform.

When playing astro‑maze, the delay is noticeable, and in a 2D action game such delays are especially apparent. Games that don’t rely on tight real‑time input might perform better. (I'm connecting from Europe, though.)

If you add support for drawing from images (such as spritesheets or tilesheets) in the future, and the client stores those images and sounds locally, the entire screen could be drawn from these assets, so no pixel data would need to be transferred, only commands like "draw tile 56 at position (x, y)."

(By the way, opening abstra.io in a German-language browser leads to https://www.abstra.io/deundefined which shows a 404 error.)

Yeah.. As people are playing and I'm watching their feedbacks it is becoming clear to me that the main source of input delay comes from the distance to the server.. the whole game is running in a single machine in SFO, so it makes total sense this bad exp in europe

I think this is inevitable unless I add some optimism/interpolation in the client

Also, thanks for the feedback! I will fix the Abstra landing page

try https://www.abstra.io/en instead

You're running this at the airport?

It's a Googleism. The datacenter is referred to by the nearest airport because airport codes make it straightforwardsto know roughly where the DC is.

For completeness sake this is not strictly a Google thing. Many companies use airport codes for their data-centers.

Cheating with AI will be possible even with server side rendering ; nvidia has released models able to learn to play - it's going to be very difficult to detect whether it's an AI or a human ; very impressive however

That’s a very different kind of cheating though. The kind of cheating this effectively makes impossible is cheating where a player has more information than they’re intended to have.

If someone makes an AI that plays the game as a good player, then it’s effectively indistinguishable from a real player who is good. If they make it super-humanly good, then it would probably be detectable anyway.

It’s still fair in the sense that all players have the same (intended) information per the game rules.

I’m also curious if an AI could process the screen feed quickly enough to compete in first-person shooter games. Seems like it would be difficult without extremely high end hardware for the foreseeable future?

There's already models specifically for things like identifying players in Counter-Strike 2, including which team they're on.

Someone has even rigged up a system like that to a TENS system to stimulate the nerves in their arm and hand to move the mouse in the correct direction and fire when the crosshair is over the enemy.

We are definitely already there.

They documented it on YouTube for us to see:

https://youtu.be/9alJwQG-Wbk

holy shit that's amazing.

I had students build this kind of thing in 2020 by screenshotting the game and processing it with a standard OpenCV pipeline. No GenAI needed.

Thank you for educating me. How does OpenCV work from the perspective of recognizing things in an image? Is there some kind of underlying model there that learns what a target looks like or not?

The way they did it, they were writing an aimbot. So the pipeline was:

- take a screenshot

- run massive skeletal detection on it to get the skeletons of any humanoids

- of those skeletons, pick a target closest to the player

- for that target, get coordinates of head node

- run a PID to control the cursor to where the head node is located

- move the camera one frame, repeat the whole process. If you can fit that pipeline to 16ms it can run in real time.

Wow, that's fascinating. Were they able to fit the whole thing inside the 16ms frame?

oh yeah, with little problem, especially with a GPU it's not hard at all.

It already exists.

If there's enough players, the server could segregate them in tiers. All superhuman players would eventually end up competing between themselves.

Reminds me of the cave X11 games. For game play I'd suggest slowing it way down.

good feedback! I'm seeing people really struggly with the control lag + speed.

I'm always biased since I test locally with no delay when developing :)

are there any affordances for prediction or replay? you could try to help network latency by having the server resimulate a small period of time roughly equivalent to the client's network delay - it's not perfect without client-side prediction but it could help

It's possible, but harder than traditional client/server paradigm since the client here is generic so the predictablity should be based on something other than heuristics

I'm thinking about simple ML to predict inputs and feedbacks. Since the amount of data generated in the streaming is massive and well structured, it looks like a possible approach