Do you have any plans release this as an app I could install on my phone?

Yes, we do! There are a couple of user-facing annoyances at the moment, one of which I mentioned in another comment (hacks to align the color and ToF images). Hopefully these API limitations are removed soon, or we find better workarounds.

If you do release it, is it possible to not depend on Google services please? I am almost a year in my degoogled life!

I would love to, but I'm not sure how yet without a major re-write. The augmented reality code we completely rely on is part of Google Play Services for AR. I definitely do understand the benefit of degoogling, so maybe it can be a community effort once I opensource it.

> Google Play Services for AR.

Google’s branding is awful.

Google Play Music All Access -> YouTube Red

Man, we're so bad at naming, that we keep joking about it... And trying to satire that with internal codenames (the ones engineers give without adult supervision)... And somehow those often seem better than the external names :(

So does this mean the app will not be available on iPhone?

It could also possibly work on iOS, but there's a catch we have not yet explored. We're relying on something called the "confidence" score in the ToF camera API, which has 8 possible values (3 bits) in Android, but from what I last saw, has only 3 possible values total in iOS. It's not clear how this lack of info exposed by the APi will affect accuracy.

This is something we'd have to separately test, and ultimately make a wholly different app for if it's feasible on iOS.

Please don't make the functionality worse just to please the de-googled hacker=news crowd.

There's no reason it has to be worse. It's good development practice to design your system to loosely couple with dependencies you have no control over. Worst case you could have two versions, and the one that supports google services works better for whatever imagined reason.

Is there a support group or discord for degooglers? I did it a month or so ago and am still dealing with the ramifications. Most recently I used App Warden to disable all the spyware libs (including Google's) shipped with WeChat. They detected this, and banned my 10 year old account with all my friends and relatives. Been struggling for two days trying to get it reactivated, cuz it will be extremely difficult to travel back to China without it. I am not blind to the irony of keeping Tencent on my phone as I remove Google.

What would be really helpful is a database of apps, similar to the WINE project, maintaining how well they work on a de-googled phone, with a table, columns being "no root", "rooted", "with microg", etc, and a WINE style rating of how well it works and what problems you'll run in to.

You would know which banking services, etc don't work well or at all and could choose your establishments with foresight.

There is a GitHub repo[1] with a very limited list last time I checked. That the best I could fined in my degoogled time. The Techlore Youtube-Channel-Community (yes, I realise the irony) also seems like a reasonably healthy place to start. There is also a bunch of telegram groups with NoGoolag[2] at the center. I would not recommend for the 70:30 mix of conspiracy and tech support but they are at least technically very knowledgeabl, and sometimes helpful, even considering their almost 'religious' technical beliefs.

[1] https://github.com/techlore/plexus

[2]https://t.me/NoGoolag

Sorry I'm not aware of anything like this. I went "all in" : I just installed GrapheneOS on my phone, Linux on my desktop and stopped using anything Google related (except YouTube that I watch in a dedicated browser).

The Techlore matrix/discord group is a great place for this.

Off-topic

What were your biggest reliance on Google services you found in this 1 year and how did you overcome it?

Interesting question. Mainly as mentioned above we needed Google Play Services for AR. Nothing else stands out at a high level.

However there is a clear bottleneck with the Google AR team for features that the community really wants. While it's understandable, perhaps the open source libraries aren't the priority or there's short staffing, there are issues like this one - https://github.com/google-ar/arcore-android-sdk/issues/153 - that have been open since Jan 2018. This one asks that we can use the smartphone flashlight simultaneously with ARCore, but there's no visibility on how close we are at the moment. This feature would have likely improved our work's performance greatly, but even in general, many other AR developers are asking for it.

Re: overcoming it, there's not much we can do in this case. We just didn't implement the feature.

I apologize Sriram I actually I meant to ask that to the user jmnicolas who asked you to for de-googled application; Yet you provided valuable information.

Long standing issues/feature requests are typical of Google's Android ecosystem, Regardless I eagerly await for your release. I think ToF based spy-camera detection could make a great addition to the women safety apps in India.

Ah, I see. Thank you!

Where do we sign up to be notified when this is released?

+1

same here

Awesome, if there's a waiting list or a website that we can join or follow to be notified of a release please share.

Sadly no project-specific page yet, but I'll definitely update on my page at https://sriramsami.com/research/ (I think there's an RSS feed?) when it's active.

It's easy to set up a mailing list. Just sign up for mailchimp, or any one of the many newsletter products like substack or tinyletters, just to collect the email addresses. Not many people regularly read RSS anymore, sadly.

Would you consider open sourcing the app as-is?

You've got a community of people who're willing to spend time polishing.

It's definitely something I'm considering. The only hard blocker now is a hashmap that checks for our test phones' unique ID and applies a fixed transform to align both cameras. Right now, any other phones will cause an immediate crash. Minimally I think we need to disable this and maybe include a small UI to let users put in the alignment parameters (just a scale + offset).

Feel free to shoot me an email (in profile) if you'd like this UI done. No need for anything else other than the source code.

It'd be a massive benefit to society to make this widely available.

Also I appreciate the interest, thank you. I'm heading back from the conference over the next few days, so I should be more free to take a look at it soon.

iOS too?

Would your system work against flat cams? https://youtu.be/BdgwO_i5p54

Thats very interesting, first time hearing of it. Not sure, really depends on what kind of reflection it generates when light hits it, and because of the unique design I'm not sure what that will be.

Does it work against something that doesn’t exist yet??

Lensless cameras have been around for a long time, the chips and masks that are completely flat is a new idea, I guess.

Single pixel cameras are lensless, for instance.

At some point if you are light-emitting you're going to be captured. May need to turn yourself into a blackbody, i.e. hide under sheets.

The next project would be creating another device to detect these spy camera detectors

Ah yes, like the radar detector wars!

> many early "stealth" radar detectors were equipped with a radar-detector-detector-detector circuit, which shuts down the main radar receiver when the detector-detector's signal is sensed, thus preventing detection by such equipment.

https://en.m.wikipedia.org/wiki/Radar_detector#Radar_detecto...

Well if the hidden camera can be shut down by turning on a hidden camera detector, then the hidden camera detector becomes a hidden camera jammer! All the better.

A hidden camera which detects ToF sensors

What are the properties of the cameras that don't get detected? Wouldn't bad actors just use the same app to check if their cameras are detectable at installation time?

So far the biggest issue is if the camera is angled oddly away from the user (basically, outside the 20 degree observable FoV). Another issue would be if they manage to install a larger camera that returns larger reflections (which we filter out).

Regarding using the app to check, I guess that applies for the existing handheld detectors as well. It's definitely something that intelligent attackers can try to plan for, but we havent tested the adversarial robustness of the system right now. That would be a very interesting direction for us as well.

But at least for the larger camera case, we can just increase one of the filtering thresholds for reflection size, which is already doable on the UI

Great work!

Do you have a link to where you can buy these types of time of flight lasers/sensors? Curious about the additional hardware cost versus sensitivity.

Thank you!

While this work operates on ToF sensors that are already present in smartphones (e.g., Samsung S20+/Ultra), a Microsoft/Azure Kinect should also be a valid option because it has a depth camera as well. It has a higher resolution and bit-depth as well.

We initially intended to compare against the Kinect, but it doesn't fit the use case (something that you can have on you at all times). However, it could be a cheap choice for a different kind of deployment (automated hidden camera detection with robots, perhaps?)

So I had no idea phones have ToF sensors these days. Do most phones have them or only the high-priced flagships? What are the actual intended uses for them?

High priced flagships have them but there are also a few midrange phones like the Huawei P30 Pro that have them too. The trend seems pretty positive towards more ToF sensors in phones, especially because Apple has had them for two iPhone Pros in a row now.

They're basically for augmented reality applications because they sense depth. Placing objects at the right size and scale in the augmented view is much easier and more accurate with the ToF sensors, for instance.

Any phone that has sub 200ms focus times has to have some sort of TOF sensor, right? The Asus Zenfone 2/3 laser, the Huawei monochrome sensor on the honor series, the portrait and/or macro sensors on the Xiaomi mi note 10, whatever the original Zenfone had, and so on.

Not all of them are lasers, which might be important.

Do you know if the Pixel 6 series supports ToF?

I don't think so. There's "laser-assisted autofocus" which might be slang for an actual ToF sensor, but sometimes that's just a single laser pulse (I.e., no 2D image that we need).

I was wondering if cameras started using anti-reflective glass would this prevent detection? Some of the glass I’ve looked up say they reflect less than 1% of light. How sensitive is your app could it detect 1% reflections? Thanks

It's lens + sensor stackup reflection. Camera sensors are much more reflective than a lens.

Yep, the combination is highly reflective. I cited another work (by a physics research group) in a comment that modelled and tested this effect comprehensively.

Would you speculate if anti reflective glass on the outside of the camera and a layer of one way tint on the inside that would allow light through but prevent it from reflective back out? I’m just thinking what we will expect in the future to sell the “undetectable spy camera”.

That's a great question. I'm not totally sure if this is possible. If it was, I think one-way mirrors would use this technology. For now, it seems like they only work because they only let 50% of the light through + there's a brightness differential between both sides. Perhaps someone more informed about this could chime in.

Yeah, there's no magic material that can violate the second law of thermodynamics, because it's the second law of thermodynamics. One-way mirrors are just partial mirrors with less light on one side than the other.

There's no such thing as a "one way tint".

https://en.wikipedia.org/wiki/One-way_mirror#Principle_of_op...

This is a simple implementation of a one way mirror.

Metamaterials have a chance at at: https://www.nist.gov/news-events/news/2014/07/new-nist-metam...

Then they'd have a signature of reflecting less than surrounding. In the long long run, probably more robust to detect the entropy generated by computation in a physical area (i.e. heat) and not the optics.

Agreed. I expect however that any easily available thermal sensing, especially on smartphones (see FLIR, etc), could be easily defeated with a strong adversary doing good thermal management.

As you say though, extra heat is a physical guarantee, so maybe a smarter technique exists to separate signal from noise in the thermal domain that I don't know of yet.

So would something like anti reflective glass for light coming into the camera then another layer of like tint that will stop light from reflecting back out if the camera? Ya I know that doesn’t exist but That is because they were not trying to do undetected as hard before. Just thinking of counter surveillance techniques.

> Just thinking of counter surveillance techniques.

Counter-counter surveillance techniques, IIUC.

Could this be applied to longer-range applications? Concert / theater cameras, camera / binocular observers, counter-sniper detection?

While the broader technique should work for those applications, the platform (smartphone ToF sensors) probably won't. Smartphone ToF sensors have a pretty limited range. We were only able to detect cameras within 1 metre of the smartphone because we're really using the hardware for something it's not intended for.

For the applications you suggest, there are some existing military-looking devices out there that use multiple lasers to find sniper scopes, for example. My basic searching shows at least https://www.ldsystems.us/product/sniper-optics-detector/#, though I'm sure there's more.

I do like this

what technology advancement would be needed to increase the detection rate and reduce the false positives?

Right, a few things would be very useful:

- Increasing the resolution of ToF cameras (right now images are around 320 x 240) --> reflections from hidden cameras can then be more detailed, whereas now it's only 1 or 2 pixels each.

- Increasing the bit-depth of ToF images - right now every pixel is only 3 bits (8 colors). It's very hard to differentiate bright hidden camera reflections from everything else, so we had to do a lot of work for that.

- API improvements in conjunction with augmented reality libraries, e.g., a) allowing Android devs to enable the flashlight when AR apps are running b) more raw access to the ToF sensor if possible

320 x 240, is this the resolution of the depth data provided by the LiDAR sensor?

Yep, exactly.

Samsung started marketing their ISOCELL Vizion 33D camera in 2020 with 640 x 480 resolution. So it's likely we'll see better ToF resolution announced in some phones in the next year or two.

Great project btw!

Thank you! Actually, my understanding when I started this project was that I would get a 640 x 480 image (IMX516 sensor). However, I could only get a 320 x 240 image from the sensor through the Android API, so that was a bit of an oddity.

The results are even more impressive with that considered!

Thank you! Hoping for higher resolutions soon.

Hm, if "ToF sensors" are LiDARs.. why does only Apple market them as LiDAR? Why are all the "3D scanning" apps only for Apple's "Pro" devices? How come no one knows that some Android devices can do the same things? I didn't know until this thread!

My understanding is that not all ToF sensors are LiDARs. There are also different types of LiDARs, e.g., some of them scan the environment and others just emit a single beam in the same one direction. I think iPhone Pro has the scanning one, but I haven't read much about it.

I assume one is active while the other is not. With two cameras you have the option of stereoscopic analysis where you have to match pixels of two cameras looking on the scene from a different angle. That uses significant cpu-time though, difficult for real time applications. Results vary because the matching isn't trivial and the difference in angle in smartphone cameras is very low.

If you have a projector, you can do more. I believe Apple uses a flash, which has a low resolution, but is perhaps less cpu-intensive and less error prone, although it has a lower resolution. That would be a real Lidar, which is an active measurement. Of course combining that with sensible stereoscopy nets better results.

All ToF systems are active.

ToF stands for "time of flight". It works exactly by measuring how long the signal takes to go from camera to object and back [1]

Stereoscopic cameras are another type of 3R camera, they are not ToF and they are not active.

Each has pros and cons.

[1] https://en.m.wikipedia.org/wiki/Time-of-flight_camera

There's a lot of marketing involved in this naming, with Samsung calling it DepthVision and Apple calling it LIDAR. There may be a difference here, however. My understanding is that Apple LIDAR is doing what we call "direct" ToF, where the round trip travel time of laser pulses is actually measured (this can be in nanoseconds). This lines up with what self driving car (and other expensive) LIDARs do.

Most other ToF sensors use "indirect" ToF, where they measure the phase difference between incoming and outgoing signals to derive distance.

However, it gets murky as cheap 2D LIDARs on say, robot vacuum cleaners, use geometric techniques to find distance (basically return angle of a reflection). I explored this in a previous work.

TLDR: I would recommend not taking any naming at face value and reading the actual datasheet or more commonly, technical marketing materials, since few ToF manufacturers that I see have a public datasheet.