I don't see how that's a problem. It's not for everyone, yes, but I like it a lot. Talking directly to the customer allows me to create exactly what the customer needs, without any convoluted process.

If that's not for you, you need at least a requirement engineer between you and the customer. That's also okay.

From my experience, getting things done quick is almost never relevant. Getting things done “cheap” is.