It's more the latter, it doesn't actually capture the image like a camera would. Instead it captures and records every single light ray and reconstructs the environment based upon that.

That would then allow you to overlay computer generated images onto basically a heightmap that has 100% accuracy to the real world.

At least that's what I got from the video.

Actually, he was right with his first guess. This isn't about capturing film with a heightmap like RGBd cameras do, it's about capturing all light that goes through a volume of space from all angles. If you can fulyl capture all light that enters a volume of space, you can accurately recreate a view from any point in that space. So, in VR I can move my head around, both by turning my head around and moving forward, backward and side to side and accurately see any view.

This does also allow adding CG elements, but it's real strength is being able to capture enough of the light coming through so that you can almost 'teleport' to where the video was recorded.

Seems roughly analogous to the holographic universe idea where all the information in a space is captured on the surface....

You're spot on. Holograms, as in the images like on a credit card, do actually record light fields.

If you want a dense, technical explanation, I wrote about this hypothetically back in April, for the same site:

http://uploadvr.com/what-the-bleep-is-lytro-doing-in-vr/

The editor informs me that HN has crushed his site, though. Should be back up soon, I hope. (Good argument for mirroring my own articles on my otherwise-defunct blog, I suppose.)

It crushed it for a bit, but now it is back and strong

uploadvr_will, you've been marked as [dead], probably due to spam filtering on new accounts. Most users won't be able to see your posts, but you should be able to contact the admins to get it fixed.

If you click the timestamp you should be taken to a page that has a "vouch" link. So non-mods can now vouch for dead posts.

Here's the announcement: https://news.ycombinator.com/item?id=10223645

And here's a follow up: https://news.ycombinator.com/item?id=10478886

Neat, I missed that! Thanks for the heads up.

I think DanBC inadvertently gave the wrong (first) link for the original announcement of this feature. It was https://news.ycombinator.com/item?id=10298512. The link Dan gave was when we announced some other stuff, that you also should know about. :)