(I'm guessing it may be as simple as there's 4 engines instead of 2?)

There's a saying in aviation that goes something like "it takes a lot of fuel to fly a lot of fuel". Even over the course of a single flight, the fuel consumption rate goes down significantly as that fuel gets consumed. You can fit more stuff in a bigger plane, but it takes more fuel to fly the 100,000th pound than it does to fly the 100th pound, so you only want the larger capacities if you know that you are going to consistently utilize them.

- A380 is heavier, per seat, than newer aircraft designs which are making use of advanced composites, new technologies that reduce the weight of wiring, etc.

- A380's engines are older models which are less efficient than the latest-generation engines fitted to the 777X and A350

- Aircraft with 4 engines have higher operating costs than 2. Additional fuel burn, maintenance expenses, etc.

Obviously some of this could be improved if Airbus were to have developed an upgraded version of the A380 (and Rolls-Royce it's engines). There was talk of both a new-engine A380neo and a stretched A380-900 at one point. But clearly, the market was judged to be too small to justify the substantial development costs.

I have read that the Boeing 747 sits at some local maxima in design space, and to get larger, the Airbus A380 compromised by having a higher wing loading. This would reduce fuel efficiency a bit.

Very few situations will call for a new A380.

Most of the traffic growth is being eaten by the smaller planes that are getting quite a bit bigger: 50 regional seaters are being replaced by 100 seaters, A319/A320 are being replaced by A320/A321, etc. Meanwhile the bigger planes are getting smaller: 747 being downsized to 777-300ER, A350-900 or even 787-9.