But, you do have a point because 'proper' tax rates for negative externalities only pay for abatement/offset of the externalities' harm. You can't count exclusively on such taxes to pay for other unrelated public goods. (If you did, you'd be overtaxing the externality, causing underproduction of the related goods, which can be just as damaging to overall welfare as the original unabated negative externalities...)
But it does tend to reduce the externality, thus reducing tax base. Which interferes with revenue collection.
The reason I singled out capital gains is that it can often lead to a genuine positive externality: capital accumulated in the hands of those best able to discriminatively lend it out (i.e. not bankers). I liked Rodgers' suggestion of taxing short-term capital gains (taxing negative effects of short-term speculation) but not long-term ones. There are probably even more finely-grained policies that can work here.
Problem: Negative externalities are supposed to go away. At which point you have no money. So you have to tax things (like income) that people just aren't going to do away with!