As I understand it, reducing unsprung weight (which is not very much for a bicycle with no suspension beyond the tires) can have an outsized effect, but actually reducing weight mostly matters for ascending. For non-competitive cycling, other factors seem like they should be much more significant.
A big one, which is banned in most competitive formats, is a fairing. This is much more effective than having a human hunker down and try to be aerodynamic. Even for an upright cargo bike (which is generally extremely heavy), a fairing in front can make a dramatic difference on level ground with no wind.
GCN have done some non-scientific experiments on their YouTube channel on this.
The biggest benefit to carbon wheels is you can make them deeper for less weight penalty (vs aluminium) which gives you a significant aero benefit.
Clipless is also an incredibly significant change, but not as much of efficiency as the others.