I can not speak to the use of conferencing for that scenario.
But, if you are allowing flow to happen after you have dialed a user in Asterisk ('g' option on the Dial command in Asterisk - http://bit.ly/575c6) and the dialed party hangs up, I believe you must be in the media stream.
Which makes sense for IVR or other media heavy applications. However, Twilio seems to get a lot of people using it for call tracking and click to call...shouldn't touch a media server unless (and I have heard of people doing this) they use media to conference two people....
But, perhaps those aren't the kinds of applications leading to scaling problems...