The biggest issue is actually food - most of the seeds grown today are engineered to depend upon fossil-fuel based chemical fertilizers, and cannot actually survive on their own without both the chemical fertilizers and a large helping of herbicides/pesticides/fungicides (many of which are also produced by the chemical industry). Break that dependency and you have to undo the Green Revolution that multiplied agricultural output by 10x. Do that and instead of the world's current food production being able to support a population of 9B if they were all willing to eat a vegetarian diet, you'd be looking at 900M, i.e. about 85% of people on earth would have to die.

I suspect that if you took away all the demand for oil from powering cars, though, there'd be plenty left at reasonable prices for growing food and maybe even flying planes. So you do need some fossil fuel production, but get rid of its use in automobiles, heating, and electricity and the remaining reserves can be allocated much more efficiently.