> Fiducials

If you mean what I think you mean (tiny marks on everything that encode information to help computers figure out what they're looking at), I agree. In particular, I've long been waiting for someone in self-driving sphere to give up on trying to crack the problem with just imaging the world as it is. In a saner world, countries would already be standardizing machine-readable markers on roads and posts and traffic signs. I'm still hoping someone will wake up and make use of this "cheat code".

I worked for a number of years on making invisible fiducials that can appear only in infrared, and am one of the chief inventors on 3M's smart code. The idea was that on any retroreflective surface you can place this fiducial sticker and relay some information. We were originally looking at approximately 64 bits of information. This isn't a ton, but we thought it a good balance of error correction, and module (the pixels in the code) size.

At least from an IR perspective, it was generally loved by nearly everyone we spoke to, but there was no takers in the end due to the IR requirement and such cameras not being everywhere yet.

Here is a link to some of our first work. That picture of a stop sign is our first prototype with hand cut special film.

https://www.businessinsider.com/3m-hides-tech-in-sides-to-he...

Nice! Thanks for sharing! Didn't realize 3M was working in this space (but then again, whenever I ask or check if 3M is working on $randomThing, it turns out they do; it's a real-world ACME).

> but there was no takers in the end due to the IR requirement and such cameras not being everywhere yet

I wonder why that's the case. What kind of IR camera it required? I was under the impression that turning a regular camera into an IR one was just a matter of digging out its IR filter and replacing it with a visible light filter. Or did it require the kind of cameras that are used in thermal imaging? Unfortunately, I don't really know what the technical and business challenges are for ubiquitous IR. Could you share some information about it?

We could tune the material to whatever wavelength we wanted, but for ours we targeted above 850nm as that was a common filter. Regarding the actual camera to IR, you are right. Even the front facing camera on a iPhone is sufficient to read them.

Many cars use front facing cameras that with minimal adjustment could read at the proper wave lengths, but one issue for a lot of vehicles right now is that the windshield has an IR filter to minimize heat and interior damage. For cameras behind the rear view mirror, the standard windshield creates an issue. Windshields with a small cut of the film would be sufficient, but they are not manufactured to my knowledge.

For general fiducials not related to these, I had hoped to put them everywhere. Think even hidden everywhere and read by phones. But at least for a while I think the read facing cameras on phones will continue to have the filter and using the phone backwards with the front facing camera is awkward.

And punk kids would be covering them over with stickers that machine read as something entirely different. :P Not as easy a thing to crack as it could be. You could cryptographically sign them (sign your signs!) and maybe encode approximate GPS coordinates to prevent them from being moved too far from their original position, but that’s a lot of data to encode in a marker.

That's an over-worried worry. Just like existing road signs which also get tampered with, they would be a redundant source of information on top of the other self driving tech that would reduce instances of a human assistant or driver having to intervene.

Pretty easy, make it a felony. Punk kids (usually) don't go around killing people.

Because killing people feels bad, but destroying objects feels like a prank (especially to those who don't understand the consequences). I don't think laws hold back punks from murder; customs do.

Forgive me, genuine question. What happens when it snows?

Same thing that happens to humans. You see parts of the signage. Or none of it, and drive on memory (and follow the behavior of other cars).

The idea behind adding special markings for computers is that you can waste a lot of time and effort trying to perfect algorithms interpreting signage optimized for human consumption, or... you can spend much less time and effort, to much better effect, setting up additional signage that's easy for computers to consume and provides information relevant to computers (which is not necessarily the same as what humans need).

Note that roads are controlled, artificial environments. There's always someone responsible for maintaining a road and signage on it. The infrastructure to deploy additional markings dedicated to self-driving cars already exist.

Yup. Or have signage at an angle that deflects snow. Or has a heater or wiper. Or as others have said, use non-visible-light fiducials.

Humans rely on road lines and signs as well and when those are covered, human capability is reduced and they have to drive slower.

For the fiducials we previously worked on, the goals was to maximize error correction for the types of occlusions commonly on signs. Because most signs are vertical, there is a lower likely hood of a ton of snow on them. The more common occlusion is an edge occlusion from either a natural road feature, or other vehicles.

For snow that generally disrupts the sign fiducial, we had a few solutions. The first is that if fiducials are dense enough, then dead reckoning may be sufficient until the next fiducial is observerd. The second is to try and build different layers of data with different error correction capabilities. One system developed could relay a low number of bits from a far distance and reasonable error correction capability. The remaining bits were then much smaller and readable from up close. The thought being that if you are able to fully resolve a fiducial in one area then assuming an apiori map of fiducials, the first 16 to 24 bits of a 64 bit code is likely enough to accurately resolve location.

Forgive me, genuine question: even in the richest countries we apparently don't have enough budget to paint new yellow lines in the middle of the roads when they get worn away - how are we going to afford something like this?

The vision of self driving going much beyond driver assists like we have today is going to die a slow death as more and more people realize it just isn't worth the CAPEX and OPEX. Humans brains are cheap in comparison.

Fiducials don’t need to be ON the road. They can be on signage or whatever. But fiducials are cheap. Just paint. (And you can, in fact, draw them by hand using a ruler, a square, and patience.)

They are error-detecting and error-correcting digital signage that gives you precision 6dof orientation “for free.” (And you can do even better... can stick a bunch of them all over a deformable object to map back the shape and deformation.... without lengthy registration and with only a single camera... unlike vidicon) And they’re easy to implement in software. Can be extremely fast (<10ms... no real limit as you can implement it in an FPGA) and fairly lightweight and can use basically any kind of digital camera (good or bad).

Are there such fiducials+libraries available today?

You can use almost any 2D matrix barcode as a fiducial, and those include error correction/detection already. Signing it would be easy.

I’m familiar with ARUCOs, but in my experience they tend to be somewhat finicky (Eg: mid-identified with a little bit of noise or unfavorable lighting), and accurate poses tend to require a lot more fiducial markers than might be considered “reasonable”.

The standard Aruco library in openCV is not the most stable fiducial tracking. But there are lots to choose from

Hello from Montreal, a city that is booming where you can't go a block without hitting a giant hole in the road or a bridge that's about to collapse. Having proper lines painted is pretty far along our hierarchy of needs. That said, I can easily imagine machine readable markers leapfrogging most of the existing infrastructure deficit.

It's easier to get money for a futuristic sounding project than for routine shoring up of infrastructure so it doesn't keep falling apart. (Just like many codebases)

machine readable can involve more than visible spectrum. Radar and light beam reflectors, non-vis spectrum strobes, radio beacons, swarm. Volvo was working on this years back, before "autonomous" - cars sharing data via wireless comms, using installed navigation markers by roads and more. There are ways, just no central will.

Google, among others, is using private (external, not owned or controlled by google) wi-fi access points as radio beacons for navigation today. Good old wardriving google. This is one factor facilitating navigation in cities when GPS is spotty. And then you have cell towers.

Centralized, standardized navigation facilitation solutions would be better, more reliable, not require mobile internet access for alignment with beacons, as they would be in a stored database in the vehicle, etc.

Or what happens if some kids prank a fiducial towards a cliff?

What if someone redraws road lines (or fakes the speed limit signs or whatever) to confuse drivers today?

With machine-readable signage based on fiducials, not only can you error-correct the signage (making it highly resistant to simple alterations and ambiguous readings), you can also encrypt it and get 6 degree of freedom vehicle vs signage pose information. All from a single camera view. That makes it MUCH more resistant to abuse than machine-learning of human-readable signage... arguably more resistant to abuse even than human readable signage is to humans.

(Also, I’ve thought of additional ways that fiducials could be resistant to such measures.)

What happens today when some kids move road cones so they point to oncoming traffic as a prank? You can have a car react the same way I did in that situation - slow down, take in the entirety of the situation ("cones say X, but that leads you into oncoming traffic") and ask the operator for directions or maneuver accordingly.

Just because some Teslas are dumb enough to drive into barriers today doesn't mean we can't continue improving the technology.

You’re actively paying huge dollar figures to experiment with your survival. Which I don’t judge even if I find it too risky for my taste. But your giant expensive experimental hunk of alloys and minerals is also a death sentence for others when it messes up.

Which is why we should use something less experimental and black-boxy than machine learning and more verifiable and robust like fiducials. (Regardless of whether we're talking about using machine vision for attempts at "self-driving," autopilot, or just driver safety features like lane-keeping or auto-braking or automatic speed limit reading.)

I mean you’d hope that self driving cars had a certain amount of error correction and weren’t just looking at one data point...

Hope is not a great thing to depend on when it comes to multi-ton fast moving objects

Natural feature tracking is cooler. I mean.... little black boxes are alright but just establishing that a bespoke 3D shape or hierarchy of shapes to orient your targeting looks less crappy and has a lot more interesting potential.

One time many many years ago I went to kinkos and printed giant fiducial squares like 48"x48" and wheat pasted them all over Austin, TX to do large at scale AR with markers. Just putting giant dongs on down town condos and stuff. Good times.

I did something similar in Europe about ten years ago but lost access to the domain I used as part of the process. That happened through no fault of my own, but it really took the wind out of my sails to run into such bad luck with one of the critical components. However it was a lot of fun working on it, and I've been thinking about this and things like it but rather in the context of "street art" that only machines can see and interpret.

I always thought coded signs that also emit some sort of signal to match would make the most sense .. and some sort of consortium to set standards.

Then you could zone off areas that are designated safe for self-driving, among other things.

It would seem like the attack vector for this technology is SUPER simple to pull off. How do you trust self-driving cars that use this in a world where graffiti exists?

But this is actually the advantage of this approach vs the alternative. Self-driving (or "autopilot") cars today try to use machine learning to read human-readable signage, but they don't have any good way to verify the values with a checksum and so are really vulnerable to graffiti (as are human drivers to some extent). It's easy to checksum a fiducial. So machine-readable signage is actually much more resistant to that kind of attack than human readable signage.

Error detecting and correcting signage would be able to correct or at least detect the error, and it’s also possible to encrypt or, um, sign the signage. None of those are terribly feasible with human readable signage.

A human can take a crowbar to a railroad track. A human can drop a brick from an overpass. But modifying a signed and error corrected fiducial is gonna be pretty tough.

What good does encryption do? All someone has to do is copy or move the sign.

I think your question was rhetorical, but there's an answer:

Because it means that modifying the sign is not useful. (Unlike, say, modifying a speed limit sign.) Also, the fiducial can encode its orientation or position (perhaps in relation to other nearby signs... and could be a hash or checksum of its position, to save space), so the vehicle would be able to know there's a mismatch and thus mark the sign as suspect/unreliable if it was in any other spot or orientation.

There are other solutions to these. But the same problem you describe occurs if someone moved a human-readable sign (but without any way to checksum).

I think fiducials are not a panacea. They are just one additional data source in what needs to be a robust sensor-fusion approach. But they make a whole bunch of stuff in machine vision a LOT easier to solve. Machine-learning approaches have the same problems but with less opportunity to address them, less robustness, and more overhead.

I supposed location and information could be added to a sign along with a certification or checksum. As long as its done in a way that doesn't require exact placement, signs found outside their location are ignored.

...something like, "Forest St. Northbound CLOSED (lat,lon) <encrypted checksum>"?

The sign could easily be tied to location, no?

FYI - This is literally no different than the current state of affairs.

https://www.extremetech.com/extreme/306346-researchers-tape-...

Yeah, in fact it'd be a LOT harder to modify a fiducial that has been checksummed/signed/encrypted.

Human readable signage is potentially much easier to attack than machine-readable signage.

There are so many simpler attacks against human-driven cars that could be pulled off even easier today, and yet, they're not pulled off.

Physical world has a much different threat model than the Internet.

It could complement but not replace a general purpose system, and anyway the bigger problem is not navigation but avoiding other cars, pedestrians and obstacles on the road.

Since your solution is based on sending someone to mark the roads, why not use the same someone to simply build a map of the road, Google Maps style with higher accuracy? I think there are a few startups doing just that now.

This won't work.

Because it's incredibly vulnerable in an adversarial world. (Human readable has a human in the loop, and so can adjust for the worst adversarial attacks. Machine readable can't)

What about attaching normal cars with "beacons" that ping every 5ms? Cheaper?

Not the same. It could complement but not replace a general purpose system.

Scania has a self driving truck convoy where a manned truck is followed by several driver less trucks using such beacons. It is not a trivial solution once you dig into it- normal wifi is too slow at highway speeds, and once the air is congested and re-transmissions occur it is even slower. Scania is using 802.11p [1] and still has backups for when connection is lost or delayed.

[1] https://www.nxp.com/products/wireless/dsrc-safety-modem/road...

You'd need, what, 50 years of enforcement before you'd have enough cars on the road to trust it, and then of course what happens if its broken? What about pedestrians?

Well, yes ofcourse, object detection will still be a feature no doubt. The vehicle must have context of where it is. But overhauling infrastructure to be machine readable is an excellent idea to implement on high risk situations like suburbia and cities where pedestrians and dogs are actual factors. The vast majority of roads and infrastructure have no pedestrians. It's just highway and freeway. It would make sense for the car to follow other cars and calibrate its position accordingly without neccesarily "seeing".

Wouldn't super high resolution longitude/latitude coordination (measuring close to a feet) help this to do it without any external tool?

True, that does give direction but not context. LAT/LONG gives no opinion on traffic speed and how to overtake when to switch lanes. My idea is that the pings are the other cars saying "I'm here, I'm here..." and the car can compute it's position relative to the other cars without having to see them. This is obviously not the main system, and more of an auxillary measure so the car can function in any condition.

> And they’ll be for pedestrians, bikes, and small subways like the London Underground uses.

Assuming such tunneling is in fact practical, I vote for burying the vehicles and letting the pedestrians have the surface...

I completely agree. That really is the ideal scenario and we should indeed make the surface safe for kids to walk around with no fear of getting run over... Banish the cars underground. However there are cases where that isn’t practical or where there are weather constraints that make tunneling desirable (for instance, in Minnesota).

For a real world example of this, check Madrid's city centre. This is exactly what happened in the last few years, and the result is amazing.

Regarding the boring company, i have seen at least one engineer's breakdown of their tech and it appears they are using off the shelf tunneling equipment and their costs are in line with and sometimes higher than several other established companies that dig smaller tunnels like the Boring Company does. It seems the only competitive advantage they have is the bully pulpit that is Elon Musk and his blind followers that can popularize the idea of small tunnels.

Except it doesn't matter if their only "innovation" is normalizing using sewer tunneling type equipment with sewer tunneling type costs. That's a MASSIVE advantage over the insane costs we have for tunneling projects in the US.

To me, seems the real problem is, why are tunnels built in the US orders of magnitude more expensive than tunnels built literally anywhere else? It's a people problem, not a technical one. There are plenty of tunnel projects around the world that are in my opinion far more valuable than shuffling around private cars (the scale of public transit through such tunnels is many orders of magnitude more impactful). But until the US gets its labor costs in check building infrastructure is poor cost-value ratio there.

Fiducials are amazing and way under utilized. I’ve worked in the space and there’s a ton of innovation to be done.

https://austingwalters.com/chromatags/

Imagine encoding virtual objects or NPCs into a fiducial without a database... basically a 3D model + actions into a piece of paper you can attach anywhere.

So I imagined it, and I came to the conclusion it's not more than a QR code.

From what I understand (which is limited) a QR code can function as a fiducial but the cutting edge of fiducials far exceed QR codes in terms of recognition speed and can be understood in terms of their relative 3D orientation to the scanner much more accurately and efficiently than QR codes.

The docs page for AprilTag[0] makes for an explanatory example,

"AprilTags are conceptually similar to QR Codes, in that they are a type of two-dimensional bar code. However, they are designed to encode far smaller data payloads (between 4 and 12 bits), allowing them to be detected more robustly and from longer ranges. Further, they are designed for high localization accuracy— you can compute the precise 3D position of the AprilTag with respect to the camera."

[0]https://april.eecs.umich.edu/software/apriltag

Something with the recognition speed and 3D orientation understanding of an AprilTag and with the information density of a QR code seems like it could be quite a useful innovation.

QR code’s are a form of fiducial markers. However, detection is far different than orientation. Also for things like AR, you need quicker detection and orientation, which QR code’s don’t afford

Yup. And Actually, you need to know the corners of the QR code pretty precisely to actually de-skew the QR tag to pull out the code from it, and it’s an easy transformation to get 6dof orientation and translation from that (although you need some idea about the camera optics, but that’s easy to calibrate with similar techniques beforehand). You’re right about about speed, though, and some fiducials are much more resistant to motion blur than QR codes are (and work with much lower resolution cameras).

Yes, I too thought this, how does a QR code not already accomplish this need?

A QR code IS a type of fiducial.

There’s an entire spectrum of fiducial types based on 2D bar code matrices. QR code is the most well known.

But 2D matrices are not as resistant to motion blur, so there are advantages to some of the other types.

You're not going to be able to encode that much data on a little stamp. Its way more reliable to use a WebVR link.

3 kilobytes (what you can store on a little fiducial QR code stamp) is a pretty large amount of data. But you don't need that much. If you just encode a 64bit identifier plus 48 bits of data encoding a bounding box in half-precision floating point for a total of 112 bits (14 bytes). Or maybe 1 byte of precision in each dimension and about 100 points defining the shape for a total of 300 bytes (and another 8 for identification).

You really can store plenty of info in a fiducial. And you can use multiple fiducials to store more if you like. This is still a pretty rich vein to mine.

I haven't seen any formats that give you that much space in a reliable way, but even then, you won't get any texture mapping, normal mapping, animation, etc etc. Why bother when you can reliably store a link and have anything you want?

Links have implied costs far greater than a few KB.

Color is tricky because it varies way too much depending on the lighting conditions, right?

> 4) ... When you can build tunnels for one or two orders of magnitude cheaper than today, cheaper than an urban surface lane ...

Didn’t Las Vegas 1.5 mi tunnel cost ~$50MM ? I don’t think 1.5mi of surface road (not considering right-of-way costs) costs that much.

It was about 1.7 miles two-way, or about $3.4 million, or ~$15 million per mile including the stations and “rolling stock.”

The tunnel in Hawthorne is about $10 million for a mile.

The advantage, of course, is in the very thing you’re not considering: right of way costs. Of course it’s cheaper to not have to tunnel at all, but there are buildings and people on the surface. And NIMBYs. So including the right of way costs, tunneling is cheaper than an urban road per lane-mile.

Tunneling projects (for larger tunnels) are about $900 million per mile in the US (can be even greater). $300 million per mile overseas. So about $10-15 million per mile is indeed about one or two orders of magnitude cheaper. Making do with smaller tunnels is a big advantage, one that I think other companies and transit agencies in the US should strongly consider as a method to address the insane infrastructure costs we have. If we have to use smaller subway trains like the London Underground uses, so what? At least we could actually get them built. We’re unlikely to saturate their capacity anyway, and I’d much rather have 4 times the routes than twice the subway train diameter.

Roads cost a ridiculous amount of money. They start at a million per mile and go up from there. Think about how many miles of road their are just in your neighborhood.

The people who complain about how much of our cities are paved have a substantial point.

But this LV tunnel is one-lane, so it should be compared to one-lane surface road, which is, like you said, is ~$1MM a mile.

No, not in urban areas.

Why wouldn't you consider right-of-way costs?

> 5) Reusable launch technology. The world still doesn’t grok this. They will. It (long term) enables access to orbit comparable to really long distance airfare or air freight.

I worked at ULA for a year about five years ago. At that time they were arguing that it wasn't going to be cost effective. Back then though there may have been one or two SpaceX landings.

since I left I haven't kept up with this debate at all. Do you know if ULA changed their stance after all the successful launches?

Not that I'm aware of. But I do know others in the industry have changed their minds, like for instance RocketLab. Peter Beck, who said that if they ever were going to make a reusable rocket he'd eat his hat, actually did announce a reusable rocket and later did actually eat (part of) his hat: https://www.youtube.com/watch?v=agqxJw5ISdk&t=188s

(Love that he actually did it haha!)

so did the ESA European Space Agency with their Themis project [1]

[1] https://www.esa.int/ESA_Multimedia/Images/2020/12/Themis

Yup, I’m a fan although disappointed they have only announced the testbed instead of an orbital rocket.

> I worked at ULA for a year about five years ago. At that time they were arguing that it wasn't going to be cost effective.

When you sell the rocket and not the launch on cost plus contract, it's indeed not cost-effective.

ULA did and has done mostly fixed price contracts. (This is different than, say, the SLS contract.)

But ULA was stuck with their approach for several reasons, only a couple of which were in their control.

It is cost effective if you charge the federal government 3-4 times more than you charge private interests.

> passenger electric flight

There are unsolved physical limitations to that with no solution in the near horizon AFAIK, the energy density of batteries is simply to low (energy per kg) for airplanes to be efficient.

> from underserved regional airports sooner than you think, doing routes and speeds comparable to High Speed Rail (and with comparable efficiency) but much more granular and without massive infrastructure

I didn't understand that, is there a limitation for using normal jet fuel planes from regional airports?

> the energy density of batteries is simply to low (energy per kg) for airplanes to be efficient.

Technically, hydrogen airplanes with fuel cells are electric and much more feasible, but I don't think that's what his post meant indeed.

Fiducials. Yes good. The more the better. But we humans cannot read them. So use UV or near infrared instead of visible spectrum. Same low-cost sensors, less territorial conflict.

It’s also possible to use fiducials that have a human-readable component (like QR code’s with a logo in the middle).

> 2) Custom silicon

I know about libresilicon [1], are there any others in this space?

[1]: https://libresilicon.com/