A guy in my lab did something similar (not 360 degrees) - basically trying to do as much autonomy as possible via a single camera. His work is really great, however he must have spent at least 2 years on building a system to detect and remove rain drops and other distortions from the front window to get results that 'aren't terrible' for night and rainy conditions. All car manufacturers spend weeks out in the desert and up in the cold of alaska to get results against extreme conditions. CV will be no exception. There's a great paper (can't find it at the moment) that gives the problem of crash detection at a stop light. During the day..no problem, but at night, or rainy, or sunset (heavy glare), etc. CV just can't do the job, but what they used instead was a pairing microphone. The sound of a crash was distinct and easy enough to detect that it was a better solution to the problem in most conditions (not all, but more than CV could solve).
I'm a cv researcher, and I love to build and design algorithms with cameras. But saying your system will "eventually scale into a X" should never be uttered lightly, even if you have proof of concept on Y.
Having said that, I do think CV will create tons of novel products (if I only knew which ones)
One of the ideas I have is a type of smart dashcam - take the footage of one or more cameras (ideally 360°), apply CV and give the driver essential real-time info like cyclists or cars coming into the blind spot, sudden changes, lane drifting etc.
The hardware is cheap and easy to work with so the project is small enough for a startup, yet it's also something you can build upon and eventually scale into a full self-driving vehicle.