That's why I never use $HOME for personal files. All of my files are in the Dropbox folder or another custom folder on my work PC. It's nonstandard, so no one litters in it besides me.