Which sadly is default/enforced behaviour. One day I downloaded something on Play Movies, and moved to another location that has Chromecast connected via a 3g access point. I expected the movie to play from the cached file, but no - it just silently downloaded a new version. Most expensive streaming session ever...
That's a fallback method, but the expected methods for most sources is that the you send the Google Cast receiver (chromecast, etc) a URL for the media, and it goes and streams it directly.