The model's size is ~173MB, depending on settings. That's not much space to have memorized every single possible combination of events, nor was our data enough to cover that either.
So, thanks a bunch, Sentdex. You are rad.
The resolution of the images output by the model is very low (what is it exactly, btw?). It's not impossible that your model has memorised at least a large part of its data.
In fact the simplest explanation of your model's output (as of much of deep neural networks for machine vision) is that it's a combination of memorisation and interpolation. There was a recent ish paper by Pedro Domingos that proposed an explation of deep learning as memorisation of exemplars similar to support vectors (if I understood it correctly - only gave it a high-level read).
It's also difficult to see from your demonstration exactly what the relation between the output and the input images are. You're showing some very simple situations in the video (go left, go right) but is that all that was in the input?
For example, I'd like to see what happens when you try to drive the car over the barrier. Was that situation in the input? And if so, how is it modelled in the output?
Finally, how do you see this having real-world applications? I don't mean necessarily right now, but let's say in 30 years time. So far, you need a fully working game engine to model a tiny part of an entire game in very low resolution and very poor detail. Do you see this as somehow being extended to creating a whole novel game from scratch? If so, how?
Edit: on memorisation, it's not necessary to memorise events, only the differences between sets of pixels in different frames. For instance, most of the background and the road stays the same during most of the "game". Again, the resolution is so low that it's not unfathomable that the model has memorised the background and the small changes to it necessary to model the input. So, it interpolates, but can it extrapolate to unseen situations that are nevertheless predicted by the physics you suggest it has learned, like driving over the barrier?
That is impressive! Less than twice the size of ResNet-50 weights. Surely that is within an order of magnitude of an equivalent Unity or GoDot game+models.
I'm sorry, how is this different than normal software engineering? There's dozens of unit/integration testing memes poking fun at specifically this (which is a mostly solvable problem in ML btw, when you use out of distribution data. Give your model a 3rd end state that represents "neither").
> id did none of that, what this model did is learn all the frames of video and their chronological order according to the input.
A better explanation is that the network knows what frame to generate given the current frame (and n previous frames) and the current user input. If it was memorizing then it'd have to generate an extremely large number of scenarios (it would exponentially grow as any given frame has k possible actions from your current frame to the next frame). If Sendex can run the game for arbitrary length and take arbitrary actions then it is a far more reasonable explanation that the model is generating the frames rather than memorizing. Apply Occam's Razor.
Edit: Sentdex said the model was ~173MB, so that is not large enough to memorize the gameplay.
Beyond rote memorization, it looks like it could be explained by saying the model appears to have a found a concept of consonance and dissonance that is bounded within the field of its inputs, and a networked grammar for interacting with the up/down/left/right inputs. Some people might find that technically trivial, but as a layman I am impressed.
The "magic" part is that the response of the network appears to be so complex relative to its inputs, but given the input is so limited from a controller, it's easy to attribute more meaning to it when it is working with a finitely bounded simulated model.
Generally I'd wonder, if the behaviour appears more complex than the stimuli, do we tend to attribute intent to it?
My Tiger repelling rock^^^^^^leopard detection model works great on all animal pictures ... until you feed it a sofa https://web.archive.org/web/20150703094328/http://rocknrolln...
>able to generalize various game logic like collision/friction with vehicles and also learns aspects of rendering such as a proper reflection of the sun on the back of the car
id did none of that, what this model did is learn all the frames of video and their chronological order according to the input.
> impossible task of "splitting a car in two" to try and solve a head-on collision.
it played back both learned versions at once, like reporting confidence of round thing being 50% ball and 50% orange.