You want a killer app for VR/AR goggle style things? You’re right this would be amazing.

Apple demoed some kind of volumetric video to the press with the Vision Pro. There was a short clip of a concert and an NBA game (Nuggets?) among other things. I heard a number of people said it was like being there.

This is a step past that. Apple’s was recorded from some kind of special camera rig (I assume), but I seriously doubt it was full volumetric video from s large number of angles. It sounded more like volumetric video if you were stuck in a (very good) seat in the venue.

I’d be curious to know just how much horsepower it takes to play these back.

That's real-time rendering though, right? Is there anything preventing it from being pre-rendered in non-real-time first? Or does it have to be rendered in real-time?

I'm not familiar with any of this at all, so I'm genuinely curious.

If the context was to use in a VR/AR headset it has to be real-time. And I guess that use-case, and the related that you interactively want to walk around a scene are two of the main use-cases

I think the best way to consider it.. is as a 3-d cloud of individually addressable pixels. The size of the cloud is dimensions of real-time rendering.

The interesting bit is that it's feasible to render in realtime. We have plenty of ways to render a 3D scene if render time is not an issue.

I thought they recorded their special videos using the Vision Pro itself, which has enough sensors to build depth maps of the scene and provide novel views within a small range from the original position.

But I am half speculating and I don't really remember. That's just the impression I remember having.

That’s a feature of the headset, but I think some of the demo videos were recorded in some other manor. I seem to remember hearing a discussion on a podcast (The Talk Show? Dithering?) where it was mentioned you could see a camera rig somehow.

Perhaps that was something you could see on TV when they accidentally showed the rig that was at the same event?

Of course it’s possible they were only using a camera rig so that no one would see the device before they were ready to invade it. Which would be very Apple like.

So I’m speculating a bit. But even if they were going to professionally record events I would think they would do better than just have someone sit there with a headset on.

I saw the image you’re referencing yesterday. The unit appeared to be a plane, briefcase sized. It had two fisheye lenses about 10cm diameter each and separated by what I expect is an average human IPD.

So I think that unit was simply capturing 180 degree stereo video. Not enough to compute volume without most of it being some ML inference.

It’s not full volumetric video. Probably can simulate a few degrees off axis but nothing like what this article shows.

I started working on something like that a couple decades ago. I figured, with all of the camera feeds at a football game, there would be plenty of views generate 3d models, even with relatively naive approaches. Then the NFL did it shortly after (2008) [1], and it didn't catch on.

It's definitely the future!

[1] 2008, https://www.cnet.com/tech/services-and-software/nfl-demos-li...

[2] Some renewed interest, in 2016: https://bleacherreport.com/articles/2659861-future-of-the-nf...

This is what they use in the latest FIFA / FC games, they call it Volumetric Data Capture, basically using video footage from real sports events to capture model and animation data for the players, allowing their unique mannerisms and movements to translate to the game. In previous iterations they would have football players in motion capture suits, but they wouldn't get all the players, plus if they did it would be in stifled, studio conditions, not their natural environment.

Anyway, not quite the same as turning a match into 3D, but definitely related.

https://gamermatters.com/ea-sports-fc-24-makes-use-of-volume...

They don't even stream the NBA games in 4k because TV networks only support 1080p. I doubt they'd buy into such an expensive technology for such a niche audience.

If they could sell it for $100 ppv they might consider it.

Actually a company has been working on this for a few years now, and I believe they are currently in production. Their focus is football/soccer I believe. I was going to do a research internship at them before I dropped it for a different one. Here it is: https://www.beyondsports.nl/

Looking at it, they heavily focus on tracking the movements of players now to replay in AR

They were demoing this last year https://www.theverge.com/2022/3/16/22982243/nets-mavericks-e...

A thought experiment I like to employ when imagining the impact of a new piece of tech: reverse the timeline. If the tech were the status quo, how would the current status quo be marketed?

If moving the camera were the norm, the current status quo would probably be marketed along the lines of "No more micro-managing camera views, arguing over playback speed or fiddling with the timeline – this next leap in technology introduces pre-edited video, where each game is preprocessed by a team of highly-skilled, professional producers, selecting the best viewing angle and playback speed so you and your friends can just sit back, relax and enjoy the game."

If the fictional press release sounds good enough, the tech probably won't hit.

I remember 30 years ago during switching to digital TV broadcasting that proponents of the tech tried to sell a future where viewers of sports events would be able to select which camera to watch. Then again, imagine watching a game with 10 friends and trying to agree over cameras...

Great point. Are you familiar with McLuhan's media tetrad?

The hypothetical status quo is a wonderful example of retrieval. Every new technology should be expected to have the effect of reemphasizing something previously obsolete.

>10 friends

Wait, you have 10 friends?

I think for routine game-watching you're 100% right. But for replaying amazing/interesting/controversial plays, this tech would an enormous improvement over being captive to the broadcast team. Everyone would love to be able to grab control and fly around and zoom in on that one power dunk, critical fumble, bad foul call, etc. on demand.

Will probably be just as boring as watching a basketball game on TV.

This is what we were building at my previous start-up, though we had a focus on outdoor sports. We had built a 3D virtual world and used GPS tracks to follow athletes (ultra-marathons, paragliders), etc.

We theorized that 2 go-pro cameras on the athlete would let us completely re-create the entire scene from all angles, and inform an AI of how to re-paint our virtual world with real-world weather environments etc.

Unfortunately, 5 years ago, everyone said I was crazy to think any of this was possible.

There is a video capture of our 3D scenes from 2017 on our old website (we were a full 3d world, not video) https://ayvri.com - the tech was acquired just over a year ago.

We did this (as a side effect) for the premier league ~2009-2012 (liquidated JUST before VR appeared, where the content worked fantastically, and then ~2014 with the moverio glasses, even better in AR)

We did live player tracking (~33 cameras) on-site at every game, and for fun rendered players fifa-style free-camera. We even did some renders (capture of realtime engine) for canal+ highlights as an experiment.

edit: my own gpgpu-only,(frag shaders :), sub-100ms, uncalibrated-cameras (footage directly from sky/match of the day) r&d a few years later, also works really well on a LookingGlass https://twitter.com/HoloSports/status/1327375694884646913 (I took this to sky sports but they said it was a bit too in-the-future)