As someone born in a former colony who voted leave, I was definitely not nostalgic for the empire. If anything I feel the EU is more nostalgic for the heights of European power than the UK - all the multipolar world stuff they love.
We are going back to a two super-power world, and China is the other super power.
According to [1], in 2020 "A third of people in the UK believe Britain’s colonies were better off for being part of an empire, a higher proportion than in any of the other major colonial powers, a global survey has revealed."
And "Brexit-supporting leave voters were more than twice as pro-empire than remainers."
Further, 39% of Leave voters vs 16% of Remain voters answered yes to "Would you like Britain to still have an empire?", while 40% of Leave voters vs. 66% of Remain voters answered "no", so in that sense you could well say it's not "beloved" by Brexiteers - given they to on the balance prefer no empire, but it's beloved by a significantly larger portion of Brexiteers.
[1] https://www.theguardian.com/world/2020/mar/11/uk-more-nostal...
(I had been under the impression that the choice on offer was between being a rather weighty part of the EU and, under the guise of "independence", playing Airstrip One to the US' Oceania, but perhaps I am mistaken?
China is playing Eastasia, of course. Not bad for 1949 speculation, even if it was based on 1920's-1930's nonfiction.
In a different fictional universe: "Always two there are")
I certainly do not see the choice the way you do. My biggest objection to the EU is that it is centralising and trying to create a new country A risky exercise without common identity. A common currency that failed to be followed by fiscal central control was a huge mistake and lead to crisis in weaker economies (and the way Greece was treated, with de facto punishment imposed in return for bailout) was a disgrace. I will admit other multilateral organisations do similar (e.g. the domestic debt default unnecessarily imposed on Sri Lanka).
The UK has been playing airstrip one for a long time anyway. In someways we should be more like the French in terms of those policies.
In general, the way the EU is headed confirms my fears. Look at the last Dutch election, for example. There is a lot of bad legislation, a lot of lobbying by corporate interests - the government of the EU is far more removed from the people than national governments (which are bad enough) are.
The sun will not set on it as long as it holds Pitcairn: https://what-if.xkcd.com/48/
Edit: although come to think of it, the British Empire so beloved of Brexiteers actually made it through the Great War (unlike Austria and Russia), hanging on right up until 1956.