Remember Apple started down the "server" road decades ago with their A/UX UNIX server (https://en.wikipedia.org/wiki/A/UX). Throughout the 1990s you could get a high-end Mac spec'd as a UNIX machine if you wanted, though I've only ever seen a handful in the wild.
The introduction of a special-purpose server was an anomaly, the Xserve had no specific predecessor, perhaps some kind of "go big or go home" effort on Apple's part.
While that system wasn't bad by the standards of the time, it couldn't compete with the likes of Dell, HP and others who offered way more flexibility on configuration and who would seemingly sell super budget low-end servers at a loss to lock people into their product line so they can squeeze them on support costs, a model Apple's allergic to.
Now there is only Mac minis for it.
You're right that the Mini is a very capable workgroup server, and providers like http://macminicolo.net/ are hosting Mac "servers" by the thousands, so it's not like they've completely given up. Intel's the only other player in this space with their NUC machines and those tend to cost as much or more.
Hopefully they'll kit out the Mini better in the 2018 iterations. That 2012 quad-code i7 variant was an exceptional unit.
Another reason they killed OS X Server maybe.