hanseul

Yes, D is depth! Sorry that it wasn't clear. The depth here comes from sensors.

As I had prior experience with RGBD cameras, for iPhones, I've learned how to use their library than the fundamentals. For example, rear depth information comes from ARKit and more specifically from ARFrame and capturedDepthData property: https://developer.apple.com/documentation/arkit/arframe/2928....

I think there are two missing things here. One is that this is not coming from a specialist camera, but from iphone cameras. Many mobile phones (including some android ones) already have hardware to capture depth. And the next is about color information. This video has no information loss in terms of color compared to ordinary videos. It's just that they are mapped into 3D space. The example you are watching has 720p color resolution.

Thank you, thank you!!

Between this app with RGBD videos and 3D reconstruction, I think in the future, there can be some software turning RGBD videos to reconstructed point cloud/mesh, so that you don't need to run a dedicated 3D reconstruction app at the time recording. Maybe I'll be able to work on it like... 3 years later...?

The codebase was originally written for a HoloLens app, so you sort of correctly guessed where it is coming from! I think this app works the best with AR headsets and it also supports AR headsets through WebXR. It's just that the AR headsets are really hard to find outside of labs...

That sounds like a cool idea! Maybe I can make this website to support phone AR, then you can place the video on top of some ruler or furniture for your measuring purposes? It's nice to hear about potential use cases and good luck with your renovation!

Thanks, I agree that what Apple is doing is pretty cool, both hardware and software-wise. About supporting devices, while the better quality rear depth cameras are only available for 12/13 pro devices, the front depth camera is included in all iphones supporting face ID (3 years old), which is sufficient for running this app!

I remember watching Minority report again last year and being really surprised watching this scene!! I was like that looks like my research project but this movie is 20 years old???

Spending several years in a VR lab made me start thinking RGBD was some common term, but for sure I'll follow your advice and use more normal words next time. And you were right that it's probably too long since the above title was few letters short the limit already!

By default, mp4 files from iphones do not include depth. The RGBD videos from telegies are custom mkv files that include depth (done by directly using libmatroska, not via ffmpeg), which the webapp can display.

Thanks a lot!! This is definitely a much more interesting video and you have a lovely cat!

Thanks for trying it with the quest and for the feedback. I really haven't thought about this issue. Yes, you are right that I have held the camera with my hand. I'll think about this and probably will need to do something equivalent to video stabilization.

Thank you!! I have no experience in deep learning yet, but I can see it being useful for this system and would like to learn about it!

I'm really sorry about this... I don't think I can fix it at the moment but I'll keep work on making the app more stable. And thanks a lot for trying the app!!!

Right, the shadowed region does make the scene look less realistic, since there is no such thing in the real world. While the lazy part of me says it's okay to leave it this way, 3D inpainting is definitely something I should look into.

Thanks a lot!! I am currently working on my dissertation, while developing this system, and kicking a ball while setting up the dissertation study kind of seemed interesting to me personally but I agree that it's not something interesting for others... Next time I'll at least find a cat!!

While that is a great which I will definitely consider for the future, this website at least supports WebXR. In case you have a VR headset, please consider trying to run this video using one!

While I cannot agree with deleting this site, which would be quite a sad thing for me, I agree that the website can be improved a lot regarding this. I am planning to make the website to download more gradually than downloading the whole video first, which will let you escape from the site without downloading that much!

Hi everyone,

My name is Hanseul Jun who is currently a graduate student working on augmented reality and telepresence systems. During my graduate studies, I have been working on code related to this app since four years ago, always wanting to bring telepresence technology outside of labs.

Hope you enjoy this! You can also share a RGBD video via a URL like this example using my app (https://apps.apple.com/us/app/telegie/id1593918560) using a iphone that contains a face ID camera or a LIDAR camera. As I have been enjoying reading comments from here and any comments would be appreciated!

You can go check out a demo room at this website (Home Exercise Demo). Inside the room, you can move around with wasd keys, mouse, or the joystick on the left-bottom side.

While the demo is a recording, it is a real-time streaming app. You can try streaming yourself if you have an iphone XR or above (this app requires a depth camera).