What do you think speed limits are for then?
In the presence of vulnerable bystanders or users of the road (e.g cyclists on city streets), limit serves as protection against fatal outcomes in case of an error. That's the only case where no one could argue against limit.
Does UK track average speeds in cities? I'm not familiar with their system. But aware that Dutch use it on highways to make everyone miserable. While their east neighbor doesn't have a default limit on highways at all while still having one of the best in the world highway safety record.
There's a couple places it's used and the only people it makes miserable are people who speed. Unlike in some places over here the speed limits are generally sensible, speeding above those limits is inherently dangerous.
[1] https://abc17news.com/news/crime/2022/01/28/numbers-rising-f...
[2] https://www.theguardian.com/us-news/2023/jan/06/us-police-ki...
Edit: I've often wondered why someone didn't put a license plate reader at every on/off ramp. You can compute speed from the elapsed time and there's no way to game/avoid/learn the positions of the cameras. Purge readings that don't generate tickets, for privacy. Wouldn't take many cameras.