They're usually used as bulk capacitance, particularly at the input side of switching regulators. Thing is, their ESR is too high to be used alone so they're usually used in parallel with an MLCC ceramic. I've had success replacing both on the input stage with an aluminum polymer electrolytic (not standard aluminum) due to their low ESR and high capacitance (and better inrush rating). I usually use ceramics on the output.

[1] http://www.ti.com/lit/an/slta055/slta055.pdf

This is a great writeup on the various types from Kemet:

[2] http://www.kemet.com/Lists/TechnicalArticles/Attachments/5/A...

Probably because when designed right (they are far more finicky then ceramic) they typically offer higher energy density.

Higher capacity you mean (for a given volume), that and also very low leakage current so if you want to make a very efficient device then tantalum might pay off. Another reason is that electrolytic caps are usually wound aluminum sheets which act as a coil. If you don't want any parasitic inductance tantalum is better in that application (better frequency response).

Tantalum caps have much worse leakage current than ceramics.

The debate for most designs is not tantalum vs. electrolytic, it's electrolytic vs. solid vs. ceramic. And ceramic usually wins.

They have lower internal inductance and resistance than other electrolytics. That is very goo for making the low-pass filters one uses on power supplies.