Building a windmill, or solar panels (not sure about geo-thermal) costs a tremendous amount of resources, and energy. So yes, except for CO2 emissions oil actually can be better for the environment. Do please remember that there is more to the environment than CO2. It takes at least 10-15 years for a windmill to pay back it's environmental costs. And for solar panels it's something like 30 years (infinity actually since they don't last that long).

This is actually a great heuristic, but pollution is only part of the story. A more general (and correct) statement would be "excluding externalities (including pollution) the cheaper one really is better for the environment".

Externalities are basically costs or benefits (to society, the environment, etc.) that aren't reflected in the price paid for an item. One really important thing to note is that externalities extend far beyond pollution and can be both positive and negative.

Pollution is one of the more visible negative externalities that get discussed, but there are plenty of others (for example, driving a car involves a host of negative externalities in addition to pollution such as increased road congestion (longer commute times, more fuel burned in traffic, more accidents)).

There are also plenty of positive externalities that often get ignored. Take windmills, solar panels and hybrid cars. The positive externalities associated with manufacturing these items include lower future manufacturing costs for such items (and lower prices and environmental costs as a result) and investment in R&D aimed at making electricity and/or transportation cheaper (both in terms of environmental and financial costs). So even though solar panels may currently be losing proposition environmentally, there are positive externalities that make buying solar panels today (and thereby moving the industry forward) a Good Thing for the environment.

In any case, there's no way to actually avoid all costs being imposed on others from your actions (waste heat, if nothing else), so it's unclear to me what's to be done about it.

Unfortunately, such a simple and effective solution is politically very unpopular, which is why we have yet to see it passed into law. People want to take care of the environment, as long as it doesn't cost them anything. Well, most people anyway.

You could even couple the tax with an offsetting decrease in other taxes, if you're worried about increasing the total tax burden.

I consider those much worse than carbon.

If a pollution tax was implemented I would give the money back as cash to every person in the country. So the money would basically move in a circle, but those who are more efficient will gain.

Well, I'm predisposed against this idea, but that sounds more like an argument for than against it. If everyone's paying in proportion to their cost to others (a big 'if'), then those who cost others less should be paid more than they pay, all else equal.

I agree if you think that global warming is uncertain, or that perhaps we actually want to increase CO2 in the atmosphere, a carbon tax would be a bad idea.