I would not my IOT lightbulb (or worse, a camera) to communicate with my neighbor's house -- there are significant availability, privacy and security concerns.
The same way, I remember seeing lots of open WiFi networks, twenty years ago... and now they are virtually gone.
Instead, it would be still centralized servers, but with more reliability -- think fiber + 5G backup at home, and multi-region/cloudfare workers approach in the cloud.
What I'd like to emphasis is what I'm suggesting will be the unwanted side-effect of this type of peer-to-peer communication.
Saving bandwidth is useful, but I see no future for de-centralized control. The control connection is cheap and allows a lot of things that would be hard with decentralized control -- health monitoring, staged rollouts, instance security updates. You can see this trend even today -- there are all service meshes like Tailscale, BeyondCorp-like distributed systems, hey, even Google Wifi -- which have central control and distributed data.
So even if something strange emerges, once it starts taking up non-trivial resources, it will be logged by central server, detected (by humans) as a bug, and killed with a routine firmware update.
These of these nodes automatically detect, connect, and communicate with the nearest node. Think of it as bluetooth vs wi-fi hybrid of the future. The aim is to make a more resilient internet by eliminating single points of failure (i.e. AWS servers down), but the end result will be some sort of rudimentary network, that will lead to "emergence" of some sort.
I just realized I've been describing "Edge computing".