In theory there is no difference between theory and practice. In practice, however...

crude simulations don't invalidate theories. i hate anti-scientific quips like that.

In theory, there is no difference between theory and practice, but in practice there is.

I'm quite sure that if there is one set of people that know this by heart then it is the military. Real world experience will bring out the trouble much faster than any amount of simulation.

Use simulations to help you design, validate your design in the real world. Then update your simulation to reflect the lessons you learned during your real world tests and iterate.

Have a look at armadillo aerospace, John Carmack is as versed as anybody in that field with simulations and STILL they find out almost every trial they do that the real world is not as clean as the simulated one.

My preferred phrasing is "The only difference between practice and theory is that, in theory, there is no difference between practice and theory."

Indeed. Just take a look at Murphy's laws of combat: http://www.murphys-laws.com/murphy/murphy-war.html

Those are great, thanks :)

I don't consider that quip anti-scientific at all. I take it as a reminder that you must always test your theories against reality, because the ways that reality and theory differ might be greater than you realized.

To address your original question, the realities of controlling an actual car are probably the hard problems. Dealing with things such as the inertia of the car and tires slipping on the road are probably both required to solve the problem, and difficult to simulate accurately. Simulations only simulate what we know about. It's often the things we didn't anticipate that make the difference between success and failure.

let me put it another way then.

can the darpa challengers code EVEN handle a simulation? if not then why waste money going ahead with the full project?

I'm sure it never occurred to anyone on these teams to try out their code in a simulated environment. I'm 100% positive their development model was to crash car after car after car until they got it working. I can only imagine how expensive some of those bug hunts got.

Come on, seriously?

ya, srsly. From what I've seen they are hacking together the hardware and then constantly updating the code to get just barely working solutions, oftentimes in the middle of the testing. If some teams are running a fairly complex car physics simulator with simulated input from each of the types of sensors they are using this would be the first I've heard of it.

30 seconds on Google:

http://www.torctech.com/company/news/jaus-standard-emerges-d...

thanks calambrac. obviously I didn't look hard enough.

that presumes an unreasonable amount of homogeneity in both the programming languages of the entrants and the sensor platforms of their vehicles. The entrants may very well do that inhouse for testing purposes, but since Darpa is merely responsible running the competition and determining if the prize award is merited, why on earth would they do something that would unduly constrain the entrants?

They've successfully navigated in the real world, so I'd say yes.

Crude simulations may or may not invalidate theories, but they're notoriously bad at weeding out ultimately inviable lines of investigation. The raw, real world is messy and unpredictable in a way a simulation can never be, and until something from the DARPA Challenge or its ilk can handle meatspace, we got no business trusting meat with it.

Not to say simulations are bad things; they're just of limited utility. Beyond that point, it does get expensive. I think that's part of the reason for the Challenge in the first place.

It's not anti-scientific, it's an old engineering adage. It's the unknown unknowns that will get you.

The issue is that

- success on a simulator does not mean being able to "scale up" to real life

- success IRL would does not imply success on some simulator

The link is particularly weak for crude simulations of multi-agent, dynamical, stochastic, partially observable, real-time worlds.

Given perfect information, like from a video game (with access to the object model, etc), the DARPA challenges aren't really that difficult. You just have to plan a path and then stay on it.

Its the noise of the real world that is infinitely challenging. You have to incorporate probabilities into your decision making, which is much more difficult. Jumping a mario around is much more AI intensive because the amount of solutions is much higher. A car is fixed to roads and has a map. Mario might know where everything is, but knowing what to do is much more difficult.

I have friend who works at NASA in sim and test. Let me assure you simulations don't have to cost an order of magnitude less when the gov is involved.

All joking aside you are talking about custom designed systems. How would you design a sim that works with whatever wonky sensor setup a team can come up with? Two isn't pre-challenge VandV up to the individual teams? They should have simed it long before it got to the competition if they were going to. The competition is to prove that it works in real life\bragging rights. It isn't nearly as cool to say my simulated robot is faster than your simulated robot.

Building the vehicles with the associated sensor and control systems were just as important as the AIs controlling I suppose.