In general, over time, as applications get more complex, the main queries you use will have more fragments. But we usually convert an entire query into a single ID (representing the entire query text, fragments and all). With each "version" of the query, fragments could be added, removed, or completely modified. Usually, your server shouldn't care what the specific fragment version is, but should be able to translate the fields the fragment asks for into a function that builds a response of that shape.
But one of the advantages of GraphQL in general (Relay included) is that you don't really need to version your API: so long as a field continues to exist in your schema, and continues to return the same type of object (even if that object adds new fields or interfaces), your old query/fragment will always receive the same shape. That may mean, over time, you "turn off" more and more fields by having them return as nulls to every client that asks for them, but you shouldn't think of it in terms of versions.
This technique of persisting the queries (and fragments) to the server at build time predates Relay - we've been using it on our iOS and Android apps since 2013.
At build time these clients submit their query strings to the server and get a small identifier in return which they can use at runtime to reference the whole query. This definitely means that old queries need to be kept around as long as the clients that use them are still active. Since iOS and Android apps seem to last forever, we're still getting traffic today from just about every version of our native apps we've ever shipped, even from 2012 and 2013. Because of this we decided to not bother with garbage collecting persisted queries, in terms of all the other data Facebook retains, persisted GraphQL queries is a grain of sand in a desert. However, with a bit more work, you could easily keep a hit counter per persisted query and go remove any persisted queries which had no recent hits.
Also, it seems like Relay Modern reintroduces API versioning, but automates it behind a compiler step. Is that a fair characterization? Does the server have to implement some kind of tracking and pruning for unused fragments, or is it expected that the fragments will accumulate at a non-threatening rate and never need pruning?