I don't know anything about Visual Studio, but I do know it's quite possible to do Android development in any text editor, although it's certainly easier in an editor that understands Java. You don't need any special IDE support, or even an IDE at all. All the project creation, building, etc. can be done quite easily from the command line.

I built my first Android app using Emacs (and the command line), because I already was somewhat familiar with Emacs and was completely unfamiliar with Eclipse. Then I switched to Eclipse, mostly to get familiar with and see what all the fuss is. Like you, I'm not a fan. I'm currently trying out Intellij, which so far seems pretty nice. I'd really recommend trying it.

IMHO, I think the hardest part of your project will be trying to get Visual Studio to support Java.