The method I chose in Windows was to use pageant (part of Putty) to load the keys. Not sure if I missed this option but you can't autoload keys so when you restart pageant you have to readd the keys (which is not terrible bad, just annoying).
Haven't tried Github For Windows yet but if it fixes the hassle of using ssh keys then I'll probably use it as my main git client for Windows.
http://jameschens.com/blog/2010/05/24/starting-pageant-durin...
Then, by all means, use Cygwin or anything that helps you, even if it doesn't come with Windows, if it isn't provided by Microsoft, doesn't look familiar and if it doesn't play well with Visual Studio. If you code for Windows you are coding against an insanely complicated set of APIs.
Compare this with Mercurial, which runs great from the command line on multiple platforms. I actually prefer the command line for SCM work, but git makes it hard.
I understand Cygwin is a layer of Linux on top of the Windows kernel, but didn't Windows have some POSIX support built in?
This seems pretty friendly to me. Is there something I'm missing?
By all means switch to using the GUI after.