I had a shared drive on virtualbox where I had the problem of the clocks not being in sync. There is also the fact that some other tools aren't necessarily well behaved and will dirty files, causing spurious rebuilds.

For the most part though, you're right!