The brushes usually don't go all the way to the sides of the vacuum, that's why many of them have an additional circular brush on the side to get into the corners and scoop the dirt to the big brushes in the middle.

No need for edges, also makes navigating through narrow things easier.

Being circular allows for zero clearance turns.

If you imagine a square robot traversing a wall and approaching a 90 degree inside corner, it can’t make the turn and would also be unable to make a perpendicular move to get more space.

That said Eufy, Makita and others make square-ish robots.

It can easily make the turn if only one quadrant was square and it didn't approach the wall with that quadrant pointing forward

Mostly circular, some have slightly asymmetric shapes like Eufy e28. The mid range and higher typically have an extending arm that handles edges or corners

Neato used to have a straight edge with 90 degree corners on both front edges. My Neato D7 is thr best robot vacuum I had. Unfortunately they shut down their cloud services, so it's useless now.