But nothing says that any given device, say a Zigbee light controller (on/off) from manufacturer X, MUST implement the standard on/off cluster for that. They could have their own manufacturer-specific profile+cluster for controlling the light, where a payload of 0x11 turns it on, 0x22 turns it off, and 0xF7 toggles the light. Or maybe you send the ASCII string "on".
Suppose that HA has implemented the functionality needed to interact with the standard on/off cluster. That functionality would work with devices that implement the standard cluster, but not with manufacturer X's device that needs the string "on". But then, HA could be updated to recognize such devices and implement the required functionality.
Assuming you're familiar with HTTP: just because two services both consume and produce JSON, doesn't mean the message bodies are interchangeable.
At the RF protocol level, yes they should be. Speaking as someone who works at a company with a few Zigbee-certified products, I can say that there are a LOT of "optional" features in the Zigbee specification, and even implementations of mandatory features can have differences that lead to incompatibility.
Manufacturer-specific profiles are also a big thing. So just be aware that manufacturer A's Zigbee "hub"/coordinator may or may not be inherently compatible with manufacturer B's sensors. But I would imagine that open-source systems like HomeAssistant deal with that.