Prompt based world building is going to be the killer app for VR. Zuck was too early.

I agree with you that this is the killer app to unlock the metaverse. When everyone can create their own spaces usage will explode like Facebook or YouTube. What makes you think he's too early though? As long as he can keep funding until that point he'll win when it happens surely?

State of art AI is progressing in that direction but we are still years away from creating a fleshed out world from prompt. Zuck funneled billions intro VR chart/Unity clone with little to show. It was too early.

> Zuck was too early.

Who will be best positioned when these killer apps start coming out? Meta has like 90% headset market share, and while Apple may release something "cool" with AR it's unlikely to be a direct threat. Microsoft is cranking along with WMR and Hololens, but they aren't leading innovation or really trying to. Valve has good hardware but can't compete with Meta due to their scale and capital.

Nobody is putting money into VR like Meta. Zuck is betting the company, and I'd probably do the same in his place. FB/IG print money but growth is substantially dead and long term prospects are dim.

> Who will be best positioned when these killer apps start coming out?

Hopefully everyone. Use OpenXR as a base and release on every headset.

You can release on every headset but 90% of your market is Oculus and they'll collect 15-30% commission on every sale.

90% of a nascent market is not a huge lead

— not sent from my blackberry.

> and while Apple may release something "cool" with AR it's unlikely to be a direct threat

2nd (or 5th) mover advantage?

Advantage of moving when the path has been revealed I think.

I'm sure Nokia will take over the market again, any minute now... any minute now...

>Valve has good hardware but can't compete with Meta due to their scale and capital.

Valve is no startup but should capital ever be the limiting factor, venture capital should be ready to finance the expansion.

However, Valve has the Instagram problem. If people don't pay for their worlds but accept them filled with branded artifacts, then Meta has that infrastructure and Valve does not.

Apple's entrance into the advertising market could be seen as a preparation for their VR products, like their gaming platform (similar to Netflix). As Meta, I wouldn't bet that Apple just creates an AR device.

Microsoft on the other hand needs a VR headset for their Xbox because Sony has one. They are already cooperating with Meta on the business headset. To me, this suggests that MS will also cooperates with Meta on the gaming headset.

If Apple manages to sell as many headsets as they sell phones then gaming consoles are dwarfed. When all is said and done, I would love to see a documentary about Sony's strategic meetings. How can they align themselves to stay relevant?

I don't think current market share is an indicator of future market share here, here innovation and non-predictable stuff could change things completely. VR is not yet popular so the potential market was not tested yet. I own Meta devices and all I can say is they have great technologies from the Hardware side. Can also say that Unity is working towards a great SDK.

In B2B: how many companies use VR headsets for meetings?

If worlds can be built from a prompt, worlds will be worthless.

Conversations with other real humans can only be built from prompts, they're not worthless.

I assume it will end up with a power-law distribution just like poetry and literature, most seen as low-value, a few as generation-defining.

Then the value is not in the created world, but in the curation.

It's going to be like those scenes in Star Trek: TNG when they give instructions to the holodeck.

> Prompt based world building

bonus:

- prompt based story for RPGs

- prompt based dungeons challenges

- synthetic voice acting for NPCs

- prompt based outfits/face for story characters

We're so much closer to these things now, the progress in the last few years has been incredible.

Retirement homes in 40 years are just going to be warehouses full of pods with people in them enjoying fulldive through infinite universes.

When you run out of money it fills with carbon dioxide, waits an hour, then dumps your body and is ready for the next one.

The problem with Meta's vision for Metaverse isn't being early, it's being bland and technically uninspired.

VRChat, and before it Second Life, are both better placed on the same journey.

Heck, even Second Life running in a browser in 2010 has a better value proposition than Meta's vision, even if the frame rate was bad enough that I recon it would cause almost universal nausea in VR: https://www.youtube.com/watch?v=cOi11vq4bxU

Great comfort for when the real one is WALL-E world! :)

It would be neat to describe features of a Minecraft world and have it generated on the fly, feels doable but I could be way off.

This was the most impressive part of the Star Trek holodeck - not the simulated matter and holographic walls, but the computer you could have a conversation with and create a custom “program”.

We seem to be heading quickly to that being a reality.

Between neural radiance fields, CLIP embedding, VR headsets, and voice interfaces, if that hasn't already been done, a tech demo that glues those things together is almost certainly going to happen in the next few months as a result of you suggesting it here.

I've thought a bit about how I would add text-based procedural generation to my virtual world engine (I would love to have an "edit a book to change the world" experience, like Myst's linking books). The hard part is the composition of smaller things into larger concepts, since you have limited data.

For example, developers often devise a system of generic water tiles that they can fit together to form any sort of river. But how do you algorithmically go from the text "I want a river that starts large, gets narrow, then turns East" to a valid map layout? The water tiles and the ways they fit together are specific to the world, so there's nowhere near enough data to train a model. You could hardcode procedural logic based on key phrases ("river" + "start large" + "get narrow" + "turn east"), but that's a lot of work for a very limited implementation.

The most promising idea I've come up with is a node-based intermediate representation. You could use general data on rivers to determine how they're shaped, then use that to translate the user's prompt into a series of nodes on a map. Then, you just need to hand-implement a system to iterate the nodes and place tiles based on whether the next node is straight ahead, turning, etc. This approach could generalize to other concepts like mountains. At that point, it's just about making the text-based stage sufficiently predictive of what the user wants instead of just being a Logo-like.

The incremental-procedural generation that impresses me most is long distance connections.

We want worlds that are too big to be completely stored, so there's conceptually a tree, where high-level features (continents? cities?) exist on the top, and low-level features (the colour of the fur of a specific citizen's pet?) towards the bottom. To get the same tree, no matter the order you expand it, is the key desirable feature of incremental procedural generation. Because then you get the feeling that it's in a sense "all there", and not just made up as you go along.

But making a tree like that is easy. The hard part is the long-distance connections. If you step down the tree to a continent, to a country, to a city, to a person... and you want that guy to have an aunt. Possibly on another continent. And you want to make it so that if you restarted with the same seed, and stepped down to the same aunt, she would still have that same nephew.

Most incremental procedural games have very little in the way of long distance connections, and it makes them lose their lustre quickly. What does it matter if there are 10000 procedurally generated cities in your world, if they don't have anything to do with each other?

A few games get by this by not being incremental, but simulating all that hard stuff up front. That's how they have aunts in Dwarf Fortress. But while it's impressive, it doesn't scale up. I think the way forward must be to figure out how to embellish (that is, expanding nodes in a seed-generated tree) just what you need, when you need it.

Language models are literally good at embellishing, but then you feel that lack of consistency and coherence, because there's no seed-generated tree at the root of it.

I think I can solve your puzzle.

First assign the number of potential aunts and nephews in each continent, city, etc top down in your tree (ensuring that the numbers are equal at the top and that you can reversibly assign a compact integer to each aunt and to each nephew). Then you need an invertible pseudorandom permutation of the right size P, i.e P(P^-1(x))=x for 0<=x<N. For nephew n the aunt is a=P(n) and you can walk down the tree to generate the details of the entity which is aunt a. If you start with the aunt you just use P^-1.

Cryptography has already developed families of functions P. For N=2^128, AES is such a family. For non power of two N, you want to look at "format-preserving encryption". For example you can use a Feistel network to permute the next larger power of two and simply iterate that permutation until you get a result in range.

I think you could extend this primitive to nonuniform graphs (aunt more likely in the same city) and more kinds of relationships. You can also use rejection sampling to condition the relationships further and probably to deal with the top down population assignment being approximate.

All that said, I think procedural generation currently runs out of creativity long before you would run out of memory to store explicit relationship indices. I think it's a lot more hopeful to get more creativity out of generative AI while keeping things grounded by explicitly representing the world, even if it "doesn't scale" to creating trillions of galaxies.

Thanks for the suggestions! I did indeed try to employ a couple of "odd" cryptographic primitives when I played around with this.

Yes, it's probably a good idea to keep track of in- and outflows, in higher nodes. The higher node could have a parameter saying something about the number of aunt-nephew relationships (or any kind of relationship) that cross continent borders. Those parameters, too, could get generated on demand.

Bit of a metaquestion but what am I reading here exactly? Is this academic research? I cant see any reference to where the work was carried out, or in what context (Cornell? Or has the paper just been submitted to them?).

It's a research paper on something called a preprint platform. It's a place almost anything fitting can be put for free - to share work before it goes through the arduous process of getting it published (side rant about publishing not meaning publishing goes here). Cornell run/host arxiv, but aren't doing heavy moderation of content (journals should check much more, arxiv will be checking more for spam/non researchy stuff).

It's a bit of work between three authors, all affiliated with Microsoft and one who is also affiliated with the University of California San Diego.

My own stuff in JavaScript relying on SantaCoder https://twitter.com/utopiah/status/1621760472461594624

just eval() all things.