I worked for a number of years on making invisible fiducials that can appear only in infrared, and am one of the chief inventors on 3M's smart code. The idea was that on any retroreflective surface you can place this fiducial sticker and relay some information. We were originally looking at approximately 64 bits of information. This isn't a ton, but we thought it a good balance of error correction, and module (the pixels in the code) size.

At least from an IR perspective, it was generally loved by nearly everyone we spoke to, but there was no takers in the end due to the IR requirement and such cameras not being everywhere yet.

Here is a link to some of our first work. That picture of a stop sign is our first prototype with hand cut special film.

https://www.businessinsider.com/3m-hides-tech-in-sides-to-he...

Nice! Thanks for sharing! Didn't realize 3M was working in this space (but then again, whenever I ask or check if 3M is working on $randomThing, it turns out they do; it's a real-world ACME).

> but there was no takers in the end due to the IR requirement and such cameras not being everywhere yet

I wonder why that's the case. What kind of IR camera it required? I was under the impression that turning a regular camera into an IR one was just a matter of digging out its IR filter and replacing it with a visible light filter. Or did it require the kind of cameras that are used in thermal imaging? Unfortunately, I don't really know what the technical and business challenges are for ubiquitous IR. Could you share some information about it?

We could tune the material to whatever wavelength we wanted, but for ours we targeted above 850nm as that was a common filter. Regarding the actual camera to IR, you are right. Even the front facing camera on a iPhone is sufficient to read them.

Many cars use front facing cameras that with minimal adjustment could read at the proper wave lengths, but one issue for a lot of vehicles right now is that the windshield has an IR filter to minimize heat and interior damage. For cameras behind the rear view mirror, the standard windshield creates an issue. Windshields with a small cut of the film would be sufficient, but they are not manufactured to my knowledge.

For general fiducials not related to these, I had hoped to put them everywhere. Think even hidden everywhere and read by phones. But at least for a while I think the read facing cameras on phones will continue to have the filter and using the phone backwards with the front facing camera is awkward.

And punk kids would be covering them over with stickers that machine read as something entirely different. :P Not as easy a thing to crack as it could be. You could cryptographically sign them (sign your signs!) and maybe encode approximate GPS coordinates to prevent them from being moved too far from their original position, but that’s a lot of data to encode in a marker.

That's an over-worried worry. Just like existing road signs which also get tampered with, they would be a redundant source of information on top of the other self driving tech that would reduce instances of a human assistant or driver having to intervene.

Pretty easy, make it a felony. Punk kids (usually) don't go around killing people.

Because killing people feels bad, but destroying objects feels like a prank (especially to those who don't understand the consequences). I don't think laws hold back punks from murder; customs do.

Forgive me, genuine question. What happens when it snows?

Same thing that happens to humans. You see parts of the signage. Or none of it, and drive on memory (and follow the behavior of other cars).

The idea behind adding special markings for computers is that you can waste a lot of time and effort trying to perfect algorithms interpreting signage optimized for human consumption, or... you can spend much less time and effort, to much better effect, setting up additional signage that's easy for computers to consume and provides information relevant to computers (which is not necessarily the same as what humans need).

Note that roads are controlled, artificial environments. There's always someone responsible for maintaining a road and signage on it. The infrastructure to deploy additional markings dedicated to self-driving cars already exist.

Yup. Or have signage at an angle that deflects snow. Or has a heater or wiper. Or as others have said, use non-visible-light fiducials.

Humans rely on road lines and signs as well and when those are covered, human capability is reduced and they have to drive slower.

For the fiducials we previously worked on, the goals was to maximize error correction for the types of occlusions commonly on signs. Because most signs are vertical, there is a lower likely hood of a ton of snow on them. The more common occlusion is an edge occlusion from either a natural road feature, or other vehicles.

For snow that generally disrupts the sign fiducial, we had a few solutions. The first is that if fiducials are dense enough, then dead reckoning may be sufficient until the next fiducial is observerd. The second is to try and build different layers of data with different error correction capabilities. One system developed could relay a low number of bits from a far distance and reasonable error correction capability. The remaining bits were then much smaller and readable from up close. The thought being that if you are able to fully resolve a fiducial in one area then assuming an apiori map of fiducials, the first 16 to 24 bits of a 64 bit code is likely enough to accurately resolve location.

Forgive me, genuine question: even in the richest countries we apparently don't have enough budget to paint new yellow lines in the middle of the roads when they get worn away - how are we going to afford something like this?

The vision of self driving going much beyond driver assists like we have today is going to die a slow death as more and more people realize it just isn't worth the CAPEX and OPEX. Humans brains are cheap in comparison.

Fiducials don’t need to be ON the road. They can be on signage or whatever. But fiducials are cheap. Just paint. (And you can, in fact, draw them by hand using a ruler, a square, and patience.)

They are error-detecting and error-correcting digital signage that gives you precision 6dof orientation “for free.” (And you can do even better... can stick a bunch of them all over a deformable object to map back the shape and deformation.... without lengthy registration and with only a single camera... unlike vidicon) And they’re easy to implement in software. Can be extremely fast (<10ms... no real limit as you can implement it in an FPGA) and fairly lightweight and can use basically any kind of digital camera (good or bad).

Are there such fiducials+libraries available today?

You can use almost any 2D matrix barcode as a fiducial, and those include error correction/detection already. Signing it would be easy.

I’m familiar with ARUCOs, but in my experience they tend to be somewhat finicky (Eg: mid-identified with a little bit of noise or unfavorable lighting), and accurate poses tend to require a lot more fiducial markers than might be considered “reasonable”.

The standard Aruco library in openCV is not the most stable fiducial tracking. But there are lots to choose from

Hello from Montreal, a city that is booming where you can't go a block without hitting a giant hole in the road or a bridge that's about to collapse. Having proper lines painted is pretty far along our hierarchy of needs. That said, I can easily imagine machine readable markers leapfrogging most of the existing infrastructure deficit.

It's easier to get money for a futuristic sounding project than for routine shoring up of infrastructure so it doesn't keep falling apart. (Just like many codebases)

machine readable can involve more than visible spectrum. Radar and light beam reflectors, non-vis spectrum strobes, radio beacons, swarm. Volvo was working on this years back, before "autonomous" - cars sharing data via wireless comms, using installed navigation markers by roads and more. There are ways, just no central will.

Google, among others, is using private (external, not owned or controlled by google) wi-fi access points as radio beacons for navigation today. Good old wardriving google. This is one factor facilitating navigation in cities when GPS is spotty. And then you have cell towers.

Centralized, standardized navigation facilitation solutions would be better, more reliable, not require mobile internet access for alignment with beacons, as they would be in a stored database in the vehicle, etc.

Or what happens if some kids prank a fiducial towards a cliff?

What if someone redraws road lines (or fakes the speed limit signs or whatever) to confuse drivers today?

With machine-readable signage based on fiducials, not only can you error-correct the signage (making it highly resistant to simple alterations and ambiguous readings), you can also encrypt it and get 6 degree of freedom vehicle vs signage pose information. All from a single camera view. That makes it MUCH more resistant to abuse than machine-learning of human-readable signage... arguably more resistant to abuse even than human readable signage is to humans.

(Also, I’ve thought of additional ways that fiducials could be resistant to such measures.)

What happens today when some kids move road cones so they point to oncoming traffic as a prank? You can have a car react the same way I did in that situation - slow down, take in the entirety of the situation ("cones say X, but that leads you into oncoming traffic") and ask the operator for directions or maneuver accordingly.

Just because some Teslas are dumb enough to drive into barriers today doesn't mean we can't continue improving the technology.

You’re actively paying huge dollar figures to experiment with your survival. Which I don’t judge even if I find it too risky for my taste. But your giant expensive experimental hunk of alloys and minerals is also a death sentence for others when it messes up.

Which is why we should use something less experimental and black-boxy than machine learning and more verifiable and robust like fiducials. (Regardless of whether we're talking about using machine vision for attempts at "self-driving," autopilot, or just driver safety features like lane-keeping or auto-braking or automatic speed limit reading.)

I mean you’d hope that self driving cars had a certain amount of error correction and weren’t just looking at one data point...

Hope is not a great thing to depend on when it comes to multi-ton fast moving objects

Natural feature tracking is cooler. I mean.... little black boxes are alright but just establishing that a bespoke 3D shape or hierarchy of shapes to orient your targeting looks less crappy and has a lot more interesting potential.

One time many many years ago I went to kinkos and printed giant fiducial squares like 48"x48" and wheat pasted them all over Austin, TX to do large at scale AR with markers. Just putting giant dongs on down town condos and stuff. Good times.

I did something similar in Europe about ten years ago but lost access to the domain I used as part of the process. That happened through no fault of my own, but it really took the wind out of my sails to run into such bad luck with one of the critical components. However it was a lot of fun working on it, and I've been thinking about this and things like it but rather in the context of "street art" that only machines can see and interpret.

I always thought coded signs that also emit some sort of signal to match would make the most sense .. and some sort of consortium to set standards.

Then you could zone off areas that are designated safe for self-driving, among other things.

It would seem like the attack vector for this technology is SUPER simple to pull off. How do you trust self-driving cars that use this in a world where graffiti exists?

But this is actually the advantage of this approach vs the alternative. Self-driving (or "autopilot") cars today try to use machine learning to read human-readable signage, but they don't have any good way to verify the values with a checksum and so are really vulnerable to graffiti (as are human drivers to some extent). It's easy to checksum a fiducial. So machine-readable signage is actually much more resistant to that kind of attack than human readable signage.

Error detecting and correcting signage would be able to correct or at least detect the error, and it’s also possible to encrypt or, um, sign the signage. None of those are terribly feasible with human readable signage.

A human can take a crowbar to a railroad track. A human can drop a brick from an overpass. But modifying a signed and error corrected fiducial is gonna be pretty tough.

What good does encryption do? All someone has to do is copy or move the sign.

I think your question was rhetorical, but there's an answer:

Because it means that modifying the sign is not useful. (Unlike, say, modifying a speed limit sign.) Also, the fiducial can encode its orientation or position (perhaps in relation to other nearby signs... and could be a hash or checksum of its position, to save space), so the vehicle would be able to know there's a mismatch and thus mark the sign as suspect/unreliable if it was in any other spot or orientation.

There are other solutions to these. But the same problem you describe occurs if someone moved a human-readable sign (but without any way to checksum).

I think fiducials are not a panacea. They are just one additional data source in what needs to be a robust sensor-fusion approach. But they make a whole bunch of stuff in machine vision a LOT easier to solve. Machine-learning approaches have the same problems but with less opportunity to address them, less robustness, and more overhead.

I supposed location and information could be added to a sign along with a certification or checksum. As long as its done in a way that doesn't require exact placement, signs found outside their location are ignored.

...something like, "Forest St. Northbound CLOSED (lat,lon) <encrypted checksum>"?

The sign could easily be tied to location, no?

FYI - This is literally no different than the current state of affairs.

https://www.extremetech.com/extreme/306346-researchers-tape-...

Yeah, in fact it'd be a LOT harder to modify a fiducial that has been checksummed/signed/encrypted.

Human readable signage is potentially much easier to attack than machine-readable signage.

There are so many simpler attacks against human-driven cars that could be pulled off even easier today, and yet, they're not pulled off.

Physical world has a much different threat model than the Internet.

It could complement but not replace a general purpose system, and anyway the bigger problem is not navigation but avoiding other cars, pedestrians and obstacles on the road.

Since your solution is based on sending someone to mark the roads, why not use the same someone to simply build a map of the road, Google Maps style with higher accuracy? I think there are a few startups doing just that now.

This won't work.

Because it's incredibly vulnerable in an adversarial world. (Human readable has a human in the loop, and so can adjust for the worst adversarial attacks. Machine readable can't)

What about attaching normal cars with "beacons" that ping every 5ms? Cheaper?

Not the same. It could complement but not replace a general purpose system.

Scania has a self driving truck convoy where a manned truck is followed by several driver less trucks using such beacons. It is not a trivial solution once you dig into it- normal wifi is too slow at highway speeds, and once the air is congested and re-transmissions occur it is even slower. Scania is using 802.11p [1] and still has backups for when connection is lost or delayed.

[1] https://www.nxp.com/products/wireless/dsrc-safety-modem/road...

You'd need, what, 50 years of enforcement before you'd have enough cars on the road to trust it, and then of course what happens if its broken? What about pedestrians?

Well, yes ofcourse, object detection will still be a feature no doubt. The vehicle must have context of where it is. But overhauling infrastructure to be machine readable is an excellent idea to implement on high risk situations like suburbia and cities where pedestrians and dogs are actual factors. The vast majority of roads and infrastructure have no pedestrians. It's just highway and freeway. It would make sense for the car to follow other cars and calibrate its position accordingly without neccesarily "seeing".

Wouldn't super high resolution longitude/latitude coordination (measuring close to a feet) help this to do it without any external tool?

True, that does give direction but not context. LAT/LONG gives no opinion on traffic speed and how to overtake when to switch lanes. My idea is that the pings are the other cars saying "I'm here, I'm here..." and the car can compute it's position relative to the other cars without having to see them. This is obviously not the main system, and more of an auxillary measure so the car can function in any condition.