Imagine that space (well space-time) it is a single elastic band and you pull it by both ends, you will have a more or less uniform tension in the middle part of the band and yet everything is receding from you. Same principle here.

Using your same diagram

vacuum <---- us -----> <----vacuum----> <-----galaxy---->