we don't know that the universe was ever small, just that it used to be a lot denser than it is today.