This is actually how I have TM set up. I ignore anything not in ~/Documents (or any other location with personal files). The other stuff is so easily replaceable that I'm not concerned if I lose it.