Without server push:
1. Client requests main page ->
2. Client receives main page <-
3. Client requests subresources ->
4. Client receives subresources <-
With server push:
1. Client requests main page ->
2. Client receives push promises and main page <-
3. Client cancels promises it doesn't need ->
4. Client receives subresources <-
The only difference is that with server push, steps 2, 3 and 4 happen in parallel, and step 3 can be omitted entirely in the event that the client doesn't need to cancel any pushes.
Without server push:
1. Client requests main page ->
2. Client receives main page <-
With server push:
1. Client requests main page ->
2. Client receives push promises and main page <-
3. Client cancels promises it doesn't need ->
Assuming the server has no way of determining which assets the client has cached (which depending on implementation may not be the case) you're of course correct. However, after step 2 the page has already fully loaded in both cases, so step 3 doesn't really slow anything down.