Seems like you could just put your test files in a tmpfs, if leaving files laying around is an issue.
You need to mount and unmount fikesystems to do that.
Take a look at the library mentioned,afero, and you'll see how nice it handles working with the file system in tests.
You can have everything in memory, and a whole new fs in each test
Or maybe your using drives over a network and randomly your tests will now fail becasue of things outside your control. Things like that.
That's why when writing tests you always want them to actually do io like that.