Hi Marcus.

It would be nice to have some idea going in for what the requirements are. - I have to admit, I don't understand why I can't preview the 3D experiences right on my computer. - Likewise, when I try the QR code on my iPhone 6 ... well, I'm hardly surprised to not see anything in 3D, but it wouldn't hurt to have a little more detail than "Scan this QR code on your iPhone or iPad."

I suppose I could mention that I've done some game development and a little bit of VR/AR, but have never heard of the USDZ file format before.

Hi there!

USDZ is the format that Apple is backing for VR/AR delivery (developed by Pixar)[1]. We're betting that it is what is going to be used in any Apple headset.

Right now you can't easily view USDZs in the browser (the format is a moving target), however Apple is making strides to make it available with the proposed Model element[2].

We were doing some USDZ --> GLTF converstion (GLTF being a format that will work in the browser), but it is jankey (technical term).

I have not tried an iPhone 6 yet. I will have to pry it from my children's hands and remove a layer of cheese and excretions before I do.

I've made a video on how to do this via the browser here[3].

TBH, the experience a heap better in the iOS App (currently in testflight) [4].

[1] https://graphics.pixar.com/usd/release/wp_usdz.html (seem to have cert issues) [2] https://github.com/WebKit/explainers/tree/main/model [3] https://usdz.app/screencasts/getting-started [4] https://testflight.apple.com/join/VteIxFd7

Oh, I should also mention, HEICs are your friend (as they have depth, and accelerometer data embedded in the file format).

Hey very cool stuff, one piece of feedback it would be good to be able to delete models on the beta iPhone app, or if we can it’s not particularly obvious how to do so.

You're right! Right now it is not possible (via the phone). It should be possible via the API or https://usdz.app

Anyway to take a model that was created and get it into CAD / modeling software? I don't know much about VR/AR but I know it is a pain in the bum to replicate an object / create a model for 3d printing.

Thank you.

We allow you to download as USDZs (and sometimes do conversions to GLTFs).

You can import both of these into Blender and export as Obj (and I'm pretty sure STLs).

You may want to use our API to do the Images ---> USDZ conversion[1].

[1] https://usdz.app/api

Excellent. Thank you for the reply.

If it's not a secret, I'd love to read some design document about the backend behind this app. What's the blackbox that takes set of images as an input and produces a 3D model as an output?

It uses the Object Capture API introduced in the latest version of macOS.

The API allows you to create 3D models from images with just a few lines of code.

https://developer.apple.com/augmented-reality/object-capture

If we're talking Swift frameworks, ModelIO (underrated) and ObjectCapture are certainly part of things.

He asked for the “blackbox that takes set of images as an input and produces a 3D model as an output.”

That black box is Object Capture API, which takes a folder of images and outputs an USDZ (or OBJ) file. Model I/O isn’t necessary.

Kudos to Apple for releasing a photogrammetry pipeline that allows web and app developers to build tools like this, with zero knowledge of 3D reconstruction required.

I’ve just been trying out the photogrammetry session, it’s super simple to use. I think Apple’s photogrammetry software isn’t as advanced as other’s, but it is free and works well small scale.

I work in the same field, making a capture app for photogrammetry software

Yeah what Apple's has is a clear lead on the time required to build objects.

I think that the photogrammetry side of things, the "give me an object now" will shift towards mobile device camera apps.

It is only serverside now due to power requirements.

If they make the api available on ios it will be a game changer, i’m hoping they’ll make it available for the m1 chipped devices

The rumours say they're putting m1 level devices in the the headset. Again the power budget is the only issue!

You're not wrong (althought we do use ModelIO a lot).

I just saw that you're from https://www.aboundlabs.com and metascan.ai Your scans look great! So how does your black box work?

We've very much built on the shoulders of giants:

  - Frontend & API (Rails + jumpstartrails.com)
  - Images --> USDZ (Swift Code on a Cloud hosted Mac)
  - USDZ --> GLTF (Swift + Custom Ruby)
  - CDN (AWS Cloudfront)

Lots of Glue:

  - AWS S3 / Cloudfront
  - AWS SQS

Pretty cool!

How are you handling spam or other kinds of content moderation? Is it easier because users don’t upload a regular text/image and need to create something custom?

There hasn't been anything humans can't handle.

If it gets out of hand, we'll look at something similar to Akismet, or an ML based solution.

Right now we're hand picking experiences to make it to the front page.

At early stage you are not going to get spam anyway. Once it grows (and i hope it does) you will get fare share of spam. As you said at that point you could check solutions like Akismet, OOPSpam.

Thanks for answering! Hand picking sounds like a really good approach, simple and effective.

So funny you should mention this, yesterday we had our "first porn" moment.

Thankfully if one throws a bunch of random images at the API and trys to make a model, the algorithm just fails.

However we're thinking it might be fun to do something w/ NSFW image detection and display a silly model...

This is really great, thanks for doing this!

Hey have you guys considered licensing an API?

We don’t think is a good long term business model (with the obvious path of functionality moving to the client).

Smart. Looking forward to this being an on device capability.

What photogrammetry pipeline are you using?

It is a stupidly long AWS based pipeline w/ the photogrammetry part is in swift.