The Xperia Compacts were almost exactly their flagship counterparts but with smaller bodies/displays (and sometimes less ROM).

Indeed, I think it's been a vicious circle of lack of demand leading to lacklustre products.

This phone does look close to flagship in terms of cpu etc, though may be missing various features - multiple cameras etc - compared to actual flagships.

I'd like a tiny flagship....

There are a lot of reasons small Android phones didn't work. One of the biggest reasons Android went to bigger form factors first was because they needed the extra battery capacity. Android was a bit of a battery hog and, CPUs on the higher end Android phones were pretty demanding. That last bit is likely the biggest reason small form factor Android died an untimely death.

I'm pretty sure it was more market driven. Everyone laughed at the Galaxy Note when it launched, including me. Then we all went and bought one.