I suspect that if you took away all the demand for oil from powering cars, though, there'd be plenty left at reasonable prices for growing food and maybe even flying planes. So you do need some fossil fuel production, but get rid of its use in automobiles, heating, and electricity and the remaining reserves can be allocated much more efficiently.
Here's a quote I found in [1]
"The macronutrients required by plants are N (Nitrogen), K (Potassium) and P (Phosphorus). Oil is hydrocarbon, made from H (Hydrogen) and C (Carbon). There are no plant nutrients in oil.
Nitrogen fertilizer (N) is made from ammonia, which in turn is manufactured from natural gas, not oil"
[1] http://peakoildebunked.blogspot.com/2005/08/28-isnt-fertiliz...
We do, 99% of plastics is made using fossil fuels. Big players are trying to shift to sustainable alternatives though, for example LEGO: https://www.lego.com/en-us/aboutus/news-room/2018/march/pfp
As far as I am aware a lot of oil is used as raw material in chemical industry, not only as energy source.