This may be an artifact of how sparse the fossil record is. If something was around for 0.5m years it's far less likely to be seen than something around for 5m years. Combined with the facts that smaller animals evolve faster, and efficient body-types are preserved. EX: Sharks.