I do exactly this myself, and it works remarkably well. I had a few conflicts I had to work around between different packages, but the updated vim package directory structure has mostly solved that problem.

EDIT: The biggest problem, for me, is that since they are all dotfiles, they are invisible to normal ls operations within the stow directory. It's a minor nit, but it can be quite annoying to run up against.