Yep, it can happen that your packet gets routed to a different DC from a prior packet. But the routers in between the client and the anycast destination will do the same thing if the environment is the same. So to get routed to a new location, you would usually need either:

* A new (usually closer) DC comes online. That will probably be your destination from now on.

* The prior DC (or a critical link on the path to it) goes down.

The crucial thing is that the client will typically be routed to the closest destination to it. In the egress case the current DC may not be the closest DC to the server it is trying to reach so the return traffic would go to the wrong place. This system of identifying a server with unique IP/port(s) means that CF's network can forward the return traffic to the correct place.