Homebrew Cask's uninstall scripts are basically a community-maintained "best guess" at to how to full uninstall each piece of software. It's generally pretty reliable, and I do use it to remove non-brew installs sometimes.

Note: I have contributed casks to Homebrew Cask before.