https://rerun.io/blog/arkitscenes-slam
I spent a bunch of time digging into the ARKitScenes dataset and built a pipeline that takes as input the raw data (arkit poses/low resolution depth maps/videos) and outputs a multiview and surface accurate gaussian splat with high resolution depths and surface normals!
https://twitter.com/pablovelagomez1/status/2082256328966422542
I want to rebuild ARKit's VIO and depth estimation in an open-source way. This is a blog post on the steps I'm taking to build this out. Mostly data work to start and choosing the right dataset to do it (in this case ARKitScenes!)Let me know if you have any questions.
From 8 -> 5 -> 4 exocentric cameras, all visualized with rerun.io. I've been working on human pose capture for robotics training data and have started making it more accessible by dropping the number of cameras used and collecting my own data to make sure I'm not overfitting to open-source datasets. I've made lots of improvements to the calibration code and ended up merging the full body...
Hands‑only multi‑view hand pose pipeline with a browser demo in Rerun. Outputs: per‑view 2D keypoints + boxes, 3D keypoints, MANO joint angles & shape (all predictions, not GT). Four stages: camera estimation → shape calibration → 2D keypoints → pose optimization.Known issues: not real‑time (~5 min for 30s/8 views on a 5090), too many cameras (target: 2 exo + 1 stereo ego), struggles with...
https://pablovela5620-egoexo-forge-viewer.hf.space
Introducing EgoExo Forge - built on top of Rerun, Gradio and Huggingface Hub (I’ll be in San Francisco July 21–29 — if you’re into robotics, egocentric AI, large-scale data collection, or just want to chat, email me at [email protected]!)In my opinion, large-scale, diverse, and high-quality data is still the largest bottleneck for generalized robotics deployment. I believe that some...
https://github.com/rerun-io/annotation-example
I built a labeling UI using Gradio and Rerun in pure Python.I want to emphasize how powerful this combo is. Rerun excels at showing complex multimodal data, and Gradio makes it super easy to spin up a ui all in pure Python quicklyHere I use1. Moge to estimate relative depth2. Birefnet to remove background + generate masks3. RTMPose to estimate human posesWith this, I'm able to build a labeling...
https://huggingface.co/spaces/pablovela5620/ego-dex-viewer
Built a Gradio app that makes it easy to visualize the recent ego-dex dataset from Apple. Often feels like it's so hard to understand what's going on in a recently released dataset without easy visualization, so I built one with rerun.
https://github.com/rerun-io/pi0-lerobot/tree/hand-kinematic-fitting
MVP of Multiview Video → Camera parameters + 3D keypoints.The basic pipeline as of right now looks like this:1. Capture – Using 4 iPhones and an Insta360 Go. iPhone videos are captured via Final Cut Pro Multicam for easy sync and the exocentric view; the Insta360 Go is used for the egocentric view.2. Sync – Custom Gradio app using two @rerundotio viewers and callbacks for easily aligning frame...
https://github.com/rerun-io/pi0-lerobot/tree/hand-kinematic-fitting
I've been trying to wrap my head around fwd/bwd kinematics for imitation learning, so I built a fully‑differentiable kinematic hand skeleton in JAX and visualized it with reruns new callback system in a Jupyter Notebook. This shows each joint angle and how it impacts the kinematic skeleton.
https://app.rerun.io/version/0.23.0/index.html?url=https://h...
Continuing with my robot training collection pipeline, one of the challenges has been obtaining calibrated cameras from sparse multi-view inputs. I previously worked with Dust3r but found its accuracy insufficient for generating reliable camera parameters. Recently, Meta released VGGT—Visual Geometry Grounded Transformer—which improves both inference speed and accuracy. With assumptions such as...
I’m developing a straightforward method to collect first-person (ego) and third-person (exo) data for robotic training. I’ve been using the HO-cap dataset to establish a baseline, and here are some updates I’ve made:* added in MANO parameters * added TSDF fused mesh from realsense camera to better visualize dataset and potentially set the stage for a gsplat integration * Built a consistent...
I ported mast3r slam to use rerun visualizer + gradio to hopefully make things easier to use.I include camera poses + pointcloud for use with nerfstudio, take a look herehttps://github.com/rerun-io/mast3r-slamhttps://x.com/pablovelagomez1/status/1898128579164082278