But also light would still take time to travel. Our view of galaxies 13bn light years away would still be of them as they were 13bn years ago in the very early universe. Again unless we happened to have zero initial velocity, we’d see all the distant galaxies clustered together in one spot in the sky.
So either the conventional view is correct and it's space that has expanded moving the galaxies further apart. This is consistent with a potentially infinite universe. Alternatively all the galaxies we can see were clustered together in a tight lump 13bn years ago and flew apart.
If the latter is the case the origin point would still be visible to us. As long as our initial random velocity was below the speed of light, we would always still be able to see light coming to us from the area of the origin point. With a random initial velocity distribution the fastest half of galaxies would be spread over a volume 4x bigger than that of the slowest half of galaxies. That means galaxy density in th direction of the origin point would be something like 4x higher that it is in directions away from the origin point. But as far as we can tell galaxy and galaxy cluster distribution is even.
Then there's the cosmic microwave background. If this is the after glow of the big bang, we wouldn't see it. All the radiation would have spread out in a shell at the speed of light. So we'd need tome other explanation for the CMB and why it happens to have all the exact characteristics we predicted for a big bang after glow.
My other argument is that if big bang didn't start in a single point then the real centerpoint of it might have never been inside the bubble of our observable universe due to speed of light and space-time expansions as the galaxies moved away.
CMB is just the image of hot matter that was close to our location 13bn years ago. Same as faint, highly redshifted images of galaxies from nearly that time. Redshift is mostly from the speed of that matter relative to us back then and spacetime expanding as it was "dragged" by the matter since then. CMB is unique in a sense that it is the image of the last time when all matter around us was so hot and dense that it was opaque. But it doesn't need any special mechanism and it's radiation is no different from our point of view than light from old galaxies.
This holds for the universe without gravity, with flat spacetime, Newtonian physics, with three, two or one spatial dimension.
Seriously, try to simulate such toy universe. Or just sit down with a piece of paper and consider variant with one spatial dimension. It's primary school math and physics. You'll get Hubble law.