Are you on Beta 10.69.2.3?
If I had to articulate my reasons:
* Judging by the videos available online, my perception is that many situations that were impossible for Tesla FSD Beta a year ago have become uneventful in recent weeks. Take a look at Chuck Cook's videos for example (I like the fact that he always highlights the failures).
* Judging again by the videos available online, my perception is that Tesla FSD Beta has encountered and had to deal with more crazy edge cases than any other system. A possible explanation for this is that for a long time Tesla FSD Beta hasn't been geofenced or restricted only to certain types of roads, like highways. You can test it anywhere in North America.
* Tesla FSD Beta currently has 160,000 individuals testing it without road restrictions. As far as I know, no other system has been exposed to similar open-ended large-scale testing.
* Occupancy networks look like a real breakthrough to me -- DNNs that predict whether each voxel in a 3D model is occupied by an object, using only video data as an input. I understood the high-level explanation of these DNNs on AI Day 2. I haven't seen anything like it from anyone else.
* Tesla's DOJO also looks like a breakthrough to me. I understood the high-level explanation of it on AI Day 2. IIRC, DOJO cabinets are 6x faster at training existing neural networks than Nvidia rigs, at 6x lower cost, so call it ~36x more efficient.
Occupancy networks: waymo has published research on this before Tesla announced this at AI day (not clear to me who got there first though https://arxiv.org/pdf/2203.03875v1.pdf)
Tesla's Dojo -> Waymo has TPUs to train on
To me all of this is outweighed by the fact that Waymo has a driverless deployment and Tesla does not. I am pretty biased because as a Tesla owner I am pretty pissed off at this point at how the false positives on the system in detecting close following are stopping my safety score from getting high enough to even be able to access the product I purchased.
But it is pretty hard to say one way or another.
I'd sum up those three points as "more data and more real-world, open-ended, large-scale testing by regular people." Big difference.
> Occupancy networks: waymo has published research on this before Tesla announced this at AI day (not clear to me who got there first though https://arxiv.org/pdf/2203.03875v1.pdf)
AFAIK, Tesla FSD Beta is the only system that has been using these DNNs for open-ended testing.
> Tesla's Dojo -> Waymo has TPUs to train on
I've trained AI models on TPUs. They're nowhere near close to 36x more efficient than Nvidia GPUs.
> I am pretty biased because as a Tesla owner I am pretty pissed off at this point at how the false positives on the system in detecting close following are stopping my safety score from getting high enough to even be able to access the product I purchased.
I understand your frustration, but I also understand why Tesla is being so strict with safety scores at this point.
Maybe you should consider that when watching YouTube videos of people using it....
Also if Tesla actually published numbers on an MlPerf benchmark, I would be more inclined to believe claims about 36x better efficiency.
https://mlcommons.org/en/training-normal-20/
The fastest times I'm seeing here for image classification and for object detection (not the same, but probably closest proxy out of the tasks benchmarked) are for TPUs.
To know who has better training technology I don't think you should be using a cost-efficiency metric, it seems to me the best thing to use would be who can train networks the fastest. Cost metrics are easy to game especially if you are the ones making the chips (Of course them making chips is cheaper than buying Nvidia chips for them once the capital investment is made). To measure who is ahead in technology, I think you have to look at who can train models the fastest, and right now as far as I can tell, TPUs are unbeat for this. (Although practically speaking it's hard to pull off these large topology things externally and there are also other caveats with ML perf related to how the training setups are optimized, but nonetheless, it's a better signal than what Elon says in a presentation :) )
Tesla fsd in its current state will either crash or do some serious fuck up if you let it unattended for a few hours or maybe less (based on the disengagements in those videos). Forget about driverless Tesla with the current fsd. Waymo has been operating driverless since 2019.
I do agree that it is progressing very nicely. Imo tesla fsd needs 2 more years and a hardware update and it will be there.
Otherwise, I agree that Tesla FSD Beta has been progressing nicely. I don't know if it will take 1, 2, or 5 years to get FSD Beta to an acceptable rate of graceful failures, but I agree it looks likely to get there before the end of the decade!
> within a factor of 100x of Waymo
What's your evidence for this? Is there anybody who has done a systematic comparison of Tesla performance in Arizon zone of Waymo?
https://www.dmv.ca.gov/portal/vehicle-industry-services/auto...
Here is a more human digestible summary:
https://thelastdriverlicenseholder.com/2022/02/09/2021-disen...
In 2021, Waymo averaged ~7,900 miles per disengagement and Cruise averaged ~41,000. In 2020, Waymo averaged ~30,000 and Cruise around 28,500.
Tesla is absent from those reports as they have deliberately declared that all of their vehicles do not even qualify as L4/5 autonomous vehicles. They have furthermore not released any 3rd party auditable metrics for any of their claims. So, from an official perspective, Tesla is infinitely worse than Waymo.
From an unofficial standpoint, we can use Youtube videos and self-reported tracking by invested fans such as here https://www.teslafsdtracker.com/
Both of those classes of unofficial metrics by positively biased groups consistently demonstrate around 10-20 miles per disengagement at max.
As Waymo averaged 8,700 last year, that makes Tesla around 400-800x worse than Waymo as of last year and around 2,000-4,000x worse than Cruise as of last year.
We can also see from the unofficial Tesla fan metrics that FSD Beta has seen no material improvement from around one year ago.
But you have seen how Tesla performs in such environments, and you aren't allowed to take your hands off the wheel.
What makes you assume Tesla has the right approach and the other have companies have to be measured against it?
All fully autonomous cars are in a different legal situation then Tesla. Tesla sells Joe Shmoe a car and then tells him he can rub FSD but he's responsible and has to remain attentive then they get info about every disengagement and (mostly) avoid legal responsibility or accidents in many cases.
Waymo is fully responsible for every accident, etc so they HAVE to proceed more cautiously or they'll lose the ability to run their cars. As someone else pointed out they often are only operated in very specific areas, and often even specific streets within a geofence. So while on the surface Waymo may have full self driving operating more effectively with less problems, they're doing so in a much more controlled environment and not getting the variety of data that Tesla has from cars disengaging Literally anywhere in the US.
i didn't sign up to be killed by some idiot tech bro testing a class project where they plumbed alexnet into the steering wheel of a 2000kg vehicle and took a couple of steps downhill
the streets are already dangerous enough for pedestrians
However, to my knowledge, Tesla is still in the "bring up" step. I think that was a mistake to pack 25 dies on a single system-on-wafer. Very hard to cool.
Do you think the minisub and Tesla ventilators also existed?
Have you considered that other companies don't make it a priority to market these things? Elon knows his audience: people who will go on message boards and talk about it. Most people don't care about the underlying AI tech.
Do you think the other companies aren't making any breakthroughs? How do they have Robotaxis then?
Your entire claimed expertise seems to come from YouTube promotional videos. Maybe take a step back from marketing hype.
I work at another autonomous car company (as a security engineer not ML related work) and I know we have a Lot of simulated situations that we run the ML against and add more from situations collected from actual driving.
If you are modeling scenarios like a game engine, a "discriminator" model isn't necessary: you just check whether a simulation doesn't result in a crash.
It’s a completely different problem space, like claiming someone built a train and therefore they can easily build self driving cars since they are both “driverless”.
Edit: and surely Waymo/Cruise could launch everywhere with performance that's lower than their current launch cities, but they choose not to. I don't think there's any compelling reason to assume their tech doesn't work outside of SF or Arizona or wherever, they just don't want to be in the news for their cars plowing someone into a highway divider or running over a pedestrian.
Sequencing your genome is still relatively expensive, Probably around $15k. 23andme does not sequence your genome. They look at specific regions looking for specific markers. This doesn't invalidate your point of how great progress has been, though.
I saw the first Moon landing.
At the time, everyone, the experts, the public, everyone, expected us to colonize the Solar System within a few decades. We expected that fusion power would be too cheap to meter and that burning fossil fuels would be a thing of the past. We expected human life expectancy would soon rise to over a century in developed countries.
Serious, respected scientists said all these things, and everyone took them for granted.
None of these things did in fact come to pass.
Humans have not ventured in person even as far as the Moon in fifty years.
Our first atomic "pile" was 80 years ago in a few months, and we still don't have a fusion reactor. The tokamak was the big fusion design 50 years ago, and it still is, and we are better at them, but we still are nowhere near actually producing real power.
Life expectancy increases stalled quite fast, and then life expectancies started to regress. Americans have lost about two years of their life, basically because of their group mistrust of medical science.
It is simply diminishing returns on the amazing discovery that is the scientific method - it is unavoidable.
I don't think that's the actual reason for the decline, it's more likely another effect of the real cause. While I'm not American, it's the same story all over the world: our diet keeps getting worse, there is plastic everywhere which breaks down into micro plastics which end up in the water we drink etc. We're preparing our food with carcinogenic utensils (everything "non stick") and all of our western societies went from a healthy worker/academic career choice into mostly just getting exploited by the established players.
There have been a lot of societal changes since the post WW2 times, and summing that up into "losing trust in medical science" is a very confusing take, especially if you consider all the medical professionals that spread outright lies to profit.
And the drug companies that sold bad medicines (knowingly!) As they judged the cost of liability to be lower then the cost of not selling the drug.
Just to be clear, I'm not saying that any of these things are causing this decline either. Our societies have just changed too much since to make any confident claims about their impact.
The 'given pace of discovery' I would read as an extrapolation using (then) available technologies, not as a scientific consensus about what would be possible using new technologies in the near future.
We must be watching different videos. And experiencing Teslas differently. I see Teslas constantly slamming on their breaks on freeways, swerving across lanes, and avoiding collisions by the narrowest margins only because their owners took control before certain death. And that's just driving around in L.A. traffic, the YouTube videos are even worse. Tesla's vaunted camera-based system still can't recognize white semis or other broad, flat obstacles that a human or radar-based system would recognize instantly.
Tesla was ahead of their competitors, several years ago. Now they're way behind, and dropping further behind with every "update" that addresses the problems that got media coverage with "solutions" that indicate brittle, manual programmer overrides rather than any sort of scalable AI-driven capability.
It doesn't matter how far ahead you were at the beginning of the race, it matters how far ahead you are at the finish line.
Part of this is the fault of Tesla's marketing, but you are wildly off mark. The cars you are seeing are Autopilot, not FSD. Most of them are even the even older, radar-based autopilot.
Tesla Vision has no issues detecting white semis crossing your path. Vehicles with radar, on the other hand, struggle with discerning those from overhead bridges, so if one appears close to a bridge, you're SOL due to whitelisting.
Tesla Vision in FSD is a much, much more developed version which has been excellent about detecting its environment, especially now with the new occupancy network. Its decision making needs work but you will notice, when watching all those videos, that detection of vehicles - even occluded ones - is not a problem at all.
Your comment about useless data is also wrong. They are experts in their field and they know exactly what type of data they need. Both Tesla and Karpathy himself have shown on multiple presentations that they focus on training unique/difficult situations because more data from perfect conditions is not useful to them anymore. They have shown exactly how they do it, and even showed the great infrastructure they've built for autolabeling.
Your claim about cruise control from competitors being equal to FSD is laughable. They don't even match Autopilot: https://www.youtube.com/watch?v=xK3NcHSH49Q&list=PLVa4b_Vn4g...
https://insideevs.com/news/616509/tesla-full-self-driving-be...
TLDR: a Tesla can't identify a box in the road. IO can finally identify people, but it still doesn't do a good job of avoiding them.
Can we say that all the self-driving optimists that were completely wrong about the last decade were shortsighted?
Also, no idea what progress in biotech has anything to do with self driving. Your argument is that because some tech advanced all other possible tech will advance, seems like a logical fallacy.
However full sequencing IS available and runs around $1000.
If I recall right the first human genome sequenced cost about $1 trillion. But we now have much better algorithms, mostly due to algorithms from works such as Knuth's AOCP.
I agree with everything you said, but chuckled at this particular part (which is very wrong): > Tesla is significantly ahead of everyone else.
Everyone in the industry knows that Tesla is nowhere near the tip of the technology. What Tesla does is _fantastic marketing_. Their whole self-driving division is just a mechanism to sell more cars.
At a high level, this is why:
- The hard thing about self driving isn't the first 95%, it's the impossibly long tail of the last 5% with unique, chaotic and rare scenarios (think, a reflective citern tank with a reflection of the back of a truck transporting stop signs, or terrible weather illusions with fog).
- Doing well on the last 5% is where most of the energy from Waymo/Cruise goes (the two leaders by quite a margin).
- Tesla is camera only. Weather alone means you can't reach critical safety because of this. Cameras don't do fog well, precipitation well, or sunsets/bad lighting well (see many Tesla crashes on freeways bc of this)
- Tesla does well on the 95% and Elon is a marketing genius, with those 2 things it's easy to convince outsiders that "Tesla is significantly ahead of everyone else".
My prediction: before the end of the decade, cruise and waymo have commoditized fleets doing things that most people today would find unbelievable. Tesla is still talking a big game but ultimately won't have permits for you to be in a Tesla with your hands off of the wheel.
edit: formatting and typo
People tend to forget just how hard the edge cases in vision are!
How often a do you encounter situations like bad fog/sun set/rain at night where it’s a total struggle to drive and you slow right down to a crawl and even then only do alright because of a ton of inference?
i think Tesla deciding to go vision only will be regarded of one of the greatest blunders is self driving history.
Pattern matching for driving is probably better, frankly. You don't have people who are stressed out, pissed off, inattentive or in a hurry doing risky stuff on the road.
The accidents caused by people with a lapse in judgement are much more memorable than the billions driving safely and preventively every day.
Current AI isn't that, but in principle it could get there.
100+ car pileups in Southern California checking in to provide a counterexample.
The patchy fog in Southern California on I-5 can go from "not too bad" to "can't see your own hood" in a matter of seconds. Radar is going to catch hazards WAY before a human will.
This leaves the question of moving to radar, but for precise resolution well ahead of the vehicle you need microwaves and a lot of power, I would guess - which reduces the vehicle's range. For all I know you might parboil passersby, too. One old Mig had a radar that would kill and roast rabbits on the runway as it took off, but that's a much different use case, of course.
No AI (reasoning) exists yet, only Machine Learning. It will take decades if not centuries.
What do you mean by "reasoning", such that there is no example of a ML system that does this?
From actual AI wiki article: “highly mathematical-statistical machine learning has dominated the field”. This will never be able to drive as good as humans.
Even if they could, a goal of self driving should be to do better than a human driver. Avoiding technology that can "see" in ways a human cannot is just short-sighted, and a huge missed opportunity.
And all that still even ignores the fact that many common environmental conditions make driving only with human eyes very unsafe. Think fog or heavy rain. A car relying only in cameras to drive in those situations will be next to useless.
Agree with your other points.
Imagine a foggy condition that causes a 50 car pile up on the highway. Which is more likely to avoid the collision, a Tesla that slowed down because it couldn't see or a Waymo/Cruise blasting down the highway at 65 mph because it's Lidar can see through the fog?
Blackmore, and Aeva can see through fog and dust that others can’t. Most sensors can see sufficiently through rain and snow.
It feels like Tesla’s main strategy is to add more data, more compute power, more simulation, and hope for “convergence”. Maybe that will work, but right now it feels like Cruise’s technology feels more mature and thought through.
As of today I can’t buy a production car with Cruise. You don’t get points for building something that’s theoretically superior but not an actual product. It’s the same story with companies like Apple that wipe the floor of wannabe hardware companies with theoretically better specs. Like Apple, Tesla actually ships.
Hilarious.
Point is I would not take anything Elon says at face value. He’s a marketer who is constantly bending the truth.
They are deploy code to cars to search for potential interesting, take lots of sample, curate and create a test and training set from that data.
Also how do you know what the Tesla compute bill is? Given their investment in GPU clusters and their own development of Dojo their cost may well have grown exponentially.
> He’s a marketer who is constantly bending the truth.
He is also a CEO of two major companies that have a good track record of achieving interesting technology and growth. So just denigrating him to a 'marketer' says more about yourself.
That asset will probably be worth far more than the self driving system.
It seems like Tesla has a huge advantage in terms of training data by leveraging a fleet of millions of vehicles.
Which is more valuable, experience or technology?
He might be right or wrong about that, but he wouldn't just ignore lidar because its currently on on the market cheaply.
I was driving down the road as normal, 4 lane divided highway that's a bit hilly. Suddenly my car starts having what I can only describe as a panic attack saying I'm running a stop sign and blaring alarms.
It was detecting a giant 40ft tall red circle sign a bit away as a stop sign...
I definitely saw a case of them overfitting their neural networks lately.
Going over a single lane bridge that's an exit ramp, the car started decoding the "other side" of the concrete barrier as oncoming traffic lanes...when there was nothing there.
Between one and two years ago, that was my perception too. But the rapid progress I've seen with Tesla FSD Beta over the past couple of years, and over the last year in particular, has forced me to change my mind. (Note: I'm talking only about Tesla's beta software. Tesla's production software is behind by dozens of versions and is without a doubt technologically inferior to Waymo and Cruise.)
Less than two years ago, I would have said FSD Beta could only deal with the "first 95%" too. Now, my perception is that FSD Beta routinely handles the first > 99% and fails only on < 1% of situations. Moreover, the failures have become more graceful -- e.g., the car will stop at intersections perceived as risky and ask the driver to confirm go-ahead by pressing the accelerator. If FSD Beta continues to improve, sooner or later it will cross the threshold at which it becomes safer than most human drivers.
That said, if I see new evidence that contradicts my perceptions, I'll change my mind again. FWIW, I'd love to see videos of Cruise and Waymo vehicles, filmed by thousands of regular consumers driving autonomously on unrestricted roads, with zero editorial input from Cruise or Waymo.
Different approaches lead to different paths to solutions, I am not convinced that either will be successful and not convinces Waymo/Cruise is ahead.
Unlike those, Tesla actually makes money and uses the technology stack in more limited forms.
Do people who work in this industry actually think they've solved 95% of driving scenarios because their software can manage driving in sunny California, Nevada, and Arizona?
Automation is actually my area. Self-droving cars are in development for more than 50 years. In the 1970s it was more analogous technology that needed a lot of space. Fact is today technology is not that much better than that. Except, at the days it was highly supervised by professionals.
Many hardware devices we have in our cars today profited from those developments. Because radar and things are now cheaply avalaible. But self-driving is still not a thing. All the improvements in AI even didn't realy helped.
What many people seems to forget AI is about to detect (learned) patterns and to make decisions based on those patterns. But AI is completely unpreditcable on patterns it can't recognise. AI will classify those into a know pattern. But this is random and as such the judgedment is random. That is the big difference to a human, which can still cope with an unknown situation.
That is the reason why many people from the industry say: real self-driving cars will take very much longer than the average publics thinks and was told by Elon, Uber and others.
But it doesn't happen in predictable ways. A particular area of technology hits a wall and plateaus all the time, for myriad reasons. In 1950, you might have thought that by 2022 we would have done a lot more with nuclear technology or supersonic airplanes than we have.
AGI is a generalization and super set of self driving cars and has a far wider impact and set of people researching it.
The two problems seem equivalently hard in that fully autonomous vehicles must be five nines (?) reliably safe. That's a ridiculously hard problem.
A lot of ideas just never pan out, despite considerable investment. Or, they get it to work, but proves to be of niche use. Progress happens elsewhere.
How can you possibly make this claim when the leader of the company you say is "ahead of everyone" said they'd have FSD years ago?
Also, FYI, watching Tesla fanatics promote FSD in edited videos isn't a good judge of the technology.
Not to mention most of the training it had in US may have little resemblance to say UK or AUS. And that is ignoring culture difference in places like India or China. I am not an expert in DNA, but I don’t understand how DNA sequencing isn’t a finite problem. Comparing to AV, in a real world with so many edge cases, you are practically looking at an infinite problem.
We have had self-driving trains and self-flying planes for decades. Despite this, train operators and airplane pilots aren't going away, and, if anything, are becoming even more important professions.
That's because they're in the "providing security" business, not in the "operating machinery" business. We pay pilots to take responsibility so passengers feel safe.
It's exactly the same for cars and trucks. The self-driving car racket is a scam, because they're not solving the actual problem society wants solved.
My favorite example is that in 1987, the scientific consensus was that it would take "at least 100 years" and possibly much longer to sequence the entire human gnome.[a] But the vast majority of it was sequenced by 2000, only 13 years later. Moreover, just two decades later, anyone could get their genome sequenced for pocket change by companies like 23andme, founded in 2006.
Having some expertise and interest in AI, I regularly watch presentations by all companies working on self-driving and also look at the videos posted by beta testers online. My perceptions contradict this article: (1) The technology is progressing faster than is generally recognized, with vehicles getting progressively better at dealing with edge cases and handling failures gracefully. (2) Judging by the videos I've watched online, Tesla is significantly ahead of everyone else.
Prediction: Before the end of the decade, this article will seem... short-sighted.
[a] https://www.nature.com/scitable/topicpage/sequencing-human-g...