News for hackers. Stuff that matters.
Re:Dobb·E: An open-source framework for learning household r... by MahiShafiullah on Wednesday November 29, 2023 @12:46PM (#38462511)
Thank you for the suggestions!

1. I'm ordering a microscope mount right now! We went with 3D printing because we could get a very good coupling between the stick and the robot mount. See this figure for a comparison: https://i.imgur.com/vWopcFB.jpeg

2. Our dataset (and dataset export code) gives the depth as numpy files and trajectory information in a JSON file with calibration and trajectory information! The r3d export format from the Record3D app we used is simply a .zip file with renamed extension so we were ok with using it. Do you have suggestion as to which cinematic format you are thinking?

Re:Dobb·E: An open-source framework for learning household r... by startupsfail on Wednesday November 29, 2023 @04:27PM (#38465559)
Yes, Record3D is an option. The problem with it, you’ll never get large scale data in the format recorded by a custom app. If you use some standard app and format that is used by general public, there is a bit higher chance. For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

In a longer term the stick is likely not needed too. Ideally you’d want a human demonstrating with regular human tool, like pliers to generalize.

This is a particular mount that I’ve used. https://www.amazon.com/Starboosa-Smartphone-Microscope-Unive...

It’s pretty solid, but it doesn’t open completely, so you need to disassemble and reassemble the stick to put it on (which is not that difficult).

Re:Dobb·E: An open-source framework for learning household r... by MahiShafiullah on Wednesday November 29, 2023 @07:16PM (#38467392)
> For the iPhone with the depth sensor cinematic format from the standard camera app seem to contain depth/RGB/traj. Maybe a candidate format. Also it’s much easier to operate standard camera app.

I agree with you there. My impression was that the cinematic format is closed, unlike .r3d which is a bunch of open format files in a .zip. Do you know whether Apple has published the specs for the cinematic video format anywhere, or if there are there good libraries for handling it?

Re:Dobb·E: An open-source framework for learning household r... by beiller on Wednesday November 29, 2023 @03:48PM (#38465088)
I think you will find openexr surprisingly well supported across many platforms.
Re:Dobb·E: An open-source framework for learning household r... by MahiShafiullah on Wednesday November 29, 2023 @04:29PM (#38465575)
OpenEXR looks very interesting! Especially if OpenEXR can compress depth data nearly as efficiently as we can compress image data by making everything a video, it would be a game changer. For example, our RGB dataset is 814 MB right now while the RGB-D dataset is 77 GB, even though depth "just" adds one new channel to our data.
Re:Dobb·E: An open-source framework for learning household r... by startupsfail on Wednesday November 29, 2023 @04:35PM (#38465656)
Yes, that’s another thing. When Apple is recording RGBD in a cinematic format, it is tiny.
Re:Dobb·E: An open-source framework for learning household r... by saulrh on Wednesday November 29, 2023 @08:35PM (#38468080)
Also look into the file formats that the robotics community use, stuff like rosbag and MCAP. These are formally more like point clouds than cinematic video captures and they won't integrate well with the video-processing side of things, but if you're dumping depths and trajectories from numpy and shuttling them to other robotics stuff then they might be a better fit than a 3d video format. In particular, if you end up adding anything other than video-like data streams - temperature, internal estimates from SLAM, etc - you'll have a much better time adding it to an mcap or rosbag than you would trying to shoehorn it into a video stream format.
Re:Dobb·E: An open-source framework for learning household r... by dhon_ on Thursday November 30, 2023 @04:35PM (#38479523)
Rosbag is just a capture of published Ros topics which can include point clouds, image/video data, imu, pose etc