You can turn off the cast notification[1]. I'm pretty sure they started doing that because you could start casting things from the Home, so they wanted a way for the user to control that from the phone after it started.

[1] https://support.google.com/chromecast/answer/7206638?hl=en