And have wild swings in load - if its steady sate load then owning the kit outright and renting space in a DC is much more cost efficient than AWS.

And bonus you can write of the depreciation cost of hardware in your accounts.

Well literally only needing one server is already a contrived use case. But if I was only in charge of one server, I'd still prefer to be in charge of one server on a platform that had AMI, CloudTrail, IAM, Amazon Inspector, GuardDuty, AWS Certificate Manager, CloudFront, AWS CLI, EBS, CloudWatch, AWS Config, ALB, Route 53, Secrets Manager, AWS WAF, and plenty of the other features that I might find perfectly useful in a one server environment. The value proposition of vanilla VPS providers is just "figure it out yourself". Deciding that a feature rich platform is worth the expense over a single product service provider is often a perfectly rational and cost effective decision.