Cloudflare mentioned they use a custom Intel SKU on their blog, and they're not as big as AWS.
Your take sounds accurate for non-FAANGs (and a couple of those). For Google, not so much. There is a lot of custom acceleration and such that isn’t available to any other customer, and a nearby commenter is right, every detail about them is locked behind a very strong NDA. I’ve heard thirdhand AWS does indeed have custom instructions in the virtualization extensions stuff, but don’t know if that’s true.
Right you are, Ken.
Depends on how many you order and how reasonable the customization is. Like, what if I wanted a 10,000 38 core Xeons with a 2.5GHz Base clock and 3.5GHz Turbo clock? Such a part doesn't exist, yet sits smack dab between two existing Models, the Xeon Platnium 8368 and 8368Q. Assuming yields are already good enough, that might be the sort of thing a CPU maker will do. (Might need more than 10k units though, IDK.)
Intel does not want to be competing with an Amazon Basics processor.
But it's likely things like different core counts, frequency performance curves, perhaps more or different memory/IO controller counts, perhaps instruction extensions of some particular interest (I'd wager BF16 was available in data center SKUs long before consumer SKUs), etc.
It's nothing radical, but if you're going to buy an awful lot of them and want something slightly custom, Intel will definitely do that for you.