For the record, I was being facetious on purpose to show how ridiculous this is.
But more to your point, the adapter would belong with the phone not with the headphones. There are potentially N number of headphones, and one phone for which the adapter will be used. So the adapter being carried with the phone instead of N adapters for N headsets makes more sense.
A little dogbone shaped white silicone thing would look a lot nicer than duct tape.
I think you already realize both of those things.