stmonty

Self-plug I have a very similar blog post title but a very different approach. Honestly, I like your idea more than my own idea. https://nostalgia.dev/posts/running-doom-inside-a-transforme...

I remember watching Star Trek: The Next Generation with my aunt and uncle as a kid. I was always more of a Star Wars fan, but they gave me a huge appreciation for Star Trek. My uncle still has a miniature replica of the USS Enterprise. Rest in peace and thanks for the fond memories.

I think seeing that thread on the front page gave me the courage to post my own post!

I think it is possible but seems quite difficult to me from this experiment. The hard part is that Pokemon has a lot of different goals, and has many strategies to complete it. You would need a way to encode these intermediary goals in some way temporally that the model needs to learn to complete in order. For example, you need to collect all eight gym badges before you can challenge the Elite Four to win.

You could compose multiple network, one with the goal of learning how to interact with the world through the screenshots. This would probably require a lot of frames of the game from many diverse locations and scenarios--as the network would need to learn how battles works, items (though technically not strictly necessary), and general dialogue and menu interactions.

Then other networks could then try to learn to encode different intermediary goals trained on a bunch of noisy runs that complete that goal. Collecting this data seems tedious and difficult though and is a whole project in itself.

So my intuition is yes, but I don't think simply making my current model bigger would get us there. I see you have experience with world models--what would you try? ;)

Dying of cancer while that young is always a tragedy. I remember looking into some of the polyhedral compilation work in LLVM as I had never heard of it before. His contributions will live on.

Author here, feel free to ask me any questions. Though I am no expert in this, just someone trying to learn and have some fun doing it.

Awesome, I have been waiting for this since you originally announced it. I also enjoyed the blog posts.

This looks awesome. I've had this same exact idea for a build system, but I haven't had the time to build it yet. Cool to see someone basically build what I had imagined!

These two videos by the YouTuber CoffeeZilla seem relevant for exposing more about the R1 and its operation:

https://www.youtube.com/watch?v=NPOHf20slZg

https://www.youtube.com/watch?v=zLvFc_24vSM