For me, it's not the cost or the ability to charge my phone, but the fact that the adapter can get lost. I can't leave it plugged in because I use my headphones with non-iPhone devices too.

I perpetually loose my headphones anyway. If I had an adapter too, I'd find myself unable to listen to anything twice as often.