If all your configs follow the XDG spec then you can turn ~/.config itself into the repo,and then use .gitignore as a whitelist:

    /*
    !git/config
    !tmux/tmux.conf

I do this and clone my repo directly to .config on a new machine. The only hold out was tmux but that's solved by installing or compiling one of the recent versions released this year (3b/3c). I'll still use manual symlinks for things like bash, but they're outliers.

Works great for me.