You'll pretty much would have had human operators over every interaction. That very much hinders the Uber-like growth of the service.
Falling back to a human operator isn't that bad either: just charge people a little bit extra to talk to a human.
One of the core features that made me use Uber was the map - I could see where the driver is going and how far away is he. Also, the app is localised into language that I can understand and I can see the price upfront without having to worry about getting scammed. Recently I had to book a taxi over the phone at the end of the world (literally - Ushuaia) in a language that I can barely speak and the experience was rather stressful in comparison with using an app.
Real-time mapping would be tricker if that's really a killer feature for you.
(Perhaps SMS plus some feature-phone-level of GPS integration.)