That's what I did and it worked well for me for the most part. I ended up keeping Gmail around for the occasional service that doesn't work well with Fastmail. Off the top of my head I've had issues with:

- Frontier

- Green Man Gaming

- Paperspace

- Rainway

- SquareTrade