Our brains do an amazing job interpreting high resolution visual data and analyzing it both spatially and temporally. Our brains then take that first analysis and apply a secondary, experiential, analysis to further interpret it into various categories relevant to the current activity.
What I’ve seen from Tesla so far indicates to me that FSD shouldn’t be enabled regardless of what sensor package they’re using, let alone based on camera data only. They need to solve their ability to accurately observes their surroundings first, especially temporally. Things shouldn’t be flashing in and out that have been clearly visible to the human eye the entire time. Additionally, this all ignores the experiential portion of driving. When most people approach something like a blind driveway or crosswalk obscured by a vehicle (a dynamic, unmapped, situation), they pay special attention and sometimes change their driving behavior.
This is true / dogma in linear / non-linear regression world, but of no real import in deep learning or Bayesian methods.
My understanding is that by focusing on fewer things (vision only), they bet to make progress faster because of the simplified organisational aspect.
Q: "Does [removing some sensors] make the perception problem harder, or easier?"
(note, this is literally what Lex asked, your restatement is misleading)
A: [paraphrasing] "Well more sensor diversity makes it harder to focus on the thing that I believe really moves the needle, so by narrowing the space of consideration, I think we'll get better results"
Karpathy might not be telling the truth, I don't know. But it's a much more credible pitch than you make it sound, because it's often true that you can deliver better by focusing on a smaller number of things. Engineering has always been about tradeoffs. Nobody is offering Karpathy infinite money plus infinite resources plus infinite time to do the job.
Again, I'm not saying Karpathy is honest or correct. I'm saying that the rephrasings in this comment and this thread are hilariously unfair.